DATA Step, Macro, Functions and more

ERROR: The SAS System stopped processing this step because of insufficient memory.

Reply
Occasional Contributor
Posts: 16

ERROR: The SAS System stopped processing this step because of insufficient memory.

Hello,

 

I have received the following error message:

 

ERROR: The SAS System stopped processing this step because of insufficient memory.
NOTE: PROCEDURE SURVEYREG used (Total process time):
real time 21.09 seconds
cpu time 21.15 seconds

 

To put this into context, I am trying to run proc survey reg clustering errors along two dimensions with ~5 million observations. The macro I am using for this is posted below. I am using SAS 9.4 on Windows 10 Pro. My computer has 32 GB of ram and I believe I have set my memory options appropriately...I right clicked my SAS 9.4 icon, chose properties, and ammended the "target" line to say 

 

"C:\Program Files\SASHome\SASFoundation\9.4\sas.exe" -CONFIG "C:\Program Files\SASHome\SASFoundation\9.4\nls\en\sasv9.cfg" -MEMSIZE 30G

 

Essentially just adding the part in bold.

 

The macro I used is listed below and I am trying to include 16 independent variables. I find it hard to imagine that 30GB of memory is not sufficient to complete this task.

 

Thanks!

 

%MACRO clus2OLS(yvar, xvars, cluster1, cluster2, dset);
/* do interesection cluster*/
proc surveyreg data=&dset; cluster &cluster1 &cluster2; model &yvar= &xvars / covb; ods output CovB = CovI; quit;
/* Do first cluster */
proc surveyreg data=&dset; cluster &cluster1; model &yvar= &xvars / covb; ods output CovB = Cov1; quit;
/* Do second cluster */
proc surveyreg data=&dset; cluster &cluster2; model &yvar= &xvars / covb; ods output CovB = Cov2 ParameterEstimates = params; quit;

/* Now get the covariances numbers created above. Calc coefs, SEs, t-stats, p-vals using COV = COV1 + COV2 - COVI*/
proc iml; reset noprint; use params;
read all var{Parameter} into varnames;
read all var _all_ into b;
use Cov1; read all var _num_ into x1;
use Cov2; read all var _num_ into x2;
use CovI; read all var _num_ into x3;

cov = x1 + x2 - x3; /* Calculate covariance matrix */
dfe = b[1,3]; stdb = sqrt(vecdiag(cov)); beta = b[,1]; t = beta/stdb; prob = 1-probf(t#t,1,dfe); /* Calc stats */

print,"Parameter estimates",,varnames beta[format=8.4] stdb[format=8.4] t[format=8.4] prob[format=8.4];

conc = beta || stdb || t || prob;
cname = {"estimates" "stderror" "tstat" "pvalue"};
create clus2dstats from conc [ colname=cname ];
append from conc;

conc = varnames;
cname = {"varnames"};
create names from conc [ colname=cname ];
append from conc;
quit;

data clus2dstats; merge names clus2dstats; run;
%MEND clus2OLS;

Trusted Advisor
Posts: 1,553

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Posted in reply to starky987

Try running with system option:  -memsize = 0;

That will enable use memory and virtual memory as much as needed.

Occasional Contributor
Posts: 16

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

How do I set a system option?

Trusted Advisor
Posts: 1,553

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Posted in reply to starky987

starky987 wrote:

How do I set a system option?


Find the -memsize option in the config file and change the value to zero

or in your sas execution:

 

"C:\Program Files\SASHome\SASFoundation\9.4\sas.exe" 

    -CONFIG "C:\Program Files\SASHome\SASFoundation\9.4\nls\en\sasv9.cfg" -MEMSIZE 0

 

Occasional Contributor
Posts: 16

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

I went ahead and made that change with no luck. I've posted the fullstimer results and I think (not sure though) it is an issue related to the OS memory from that log?

 

I'm not entirely sure, I just know its not possible that this file takes up more than the 32G of ram available.

Super User
Posts: 7,758

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Posted in reply to starky987

According to the fullstimer log, you had less that 1GB memory in use. Could it be that there is a system setting (like the UNIX ulimit) in place that limits the resources available for your SAS process?

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Super User
Posts: 3,250

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Posted in reply to starky987

Run your program with the SAS option FULLSTIMER and the SAS log will report how much memory is being used in each step.

Occasional Contributor
Posts: 16

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

[ Edited ]
 
Occasional Contributor
Posts: 16

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Posted in reply to starky987

I received the following information from fulltimer

 

NOTE: PROCEDURE SURVEYREG used (Total process time):
real time 21.64 seconds
user cpu time 19.18 seconds
system cpu time 2.48 seconds
memory 572712.68k
OS Memory 652840.00k
Timestamp 02/16/2017 03:26:20 PM
Step Count 73 Switch Count 0

Frequent Contributor
Posts: 80

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Posted in reply to starky987

It doesn't help you solve this with PROC SURVEYREG, but I've had a similar issue in the past and was not able to resolve through the suggested methods. I think they work in the right environment. I'm using a server-delivered SAS and therefore do not have complete control over how much memory (or at least, that's my working theory).

 

I've ended up using a different approach: Create a random number as a variable in that data set, and then do a selection off that. So for instance, create a random number from 0 to 1, then only output when the value is less than 0.05 (5% sample). 

Occasional Contributor
Posts: 16

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Posted in reply to starky987
I am afraid that may be the case Kurt but I don't know why or how to fix it. I am running Windows 10 Pro on a personal machine and am the only user. Any thoughts?
Super User
Posts: 7,758

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Posted in reply to starky987

I have to admit that my Windows admin skills are virtually nonexistent (compared to my AIX admin skills). Even googling did not provide me with a hint.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Ask a Question
Discussion stats
  • 11 replies
  • 2214 views
  • 0 likes
  • 5 in conversation