topic Re: ERROR: The SAS System stopped processing this step because of insufficient memory. in SAS Programming

ERROR: The SAS System stopped processing this step because of insufficient memory.

starky987 — Thu, 16 Feb 2017 19:53:50 GMT

Hello,

I have received the following error message:

ERROR: The SAS System stopped processing this step because of insufficient memory.
NOTE: PROCEDURE SURVEYREG used (Total process time):
real time 21.09 seconds
cpu time 21.15 seconds

To put this into context, I am trying to run proc survey reg clustering errors along two dimensions with ~5 million observations. The macro I am using for this is posted below. I am using SAS 9.4 on Windows 10 Pro. My computer has 32 GB of ram and I believe I have set my memory options appropriately...I right clicked my SAS 9.4 icon, chose properties, and ammended the "target" line to say

"C:\Program Files\SASHome\SASFoundation\9.4\sas.exe" -CONFIG "C:\Program Files\SASHome\SASFoundation\9.4\nls\en\sasv9.cfg" -MEMSIZE 30G

Essentially just adding the part in bold.

The macro I used is listed below and I am trying to include 16 independent variables. I find it hard to imagine that 30GB of memory is not sufficient to complete this task.

Thanks!

%MACRO clus2OLS(yvar, xvars, cluster1, cluster2, dset);
/* do interesection cluster*/
proc surveyreg data=&dset; cluster &cluster1 &cluster2; model &yvar= &xvars / covb; ods output CovB = CovI; quit;
/* Do first cluster */
proc surveyreg data=&dset; cluster &cluster1; model &yvar= &xvars / covb; ods output CovB = Cov1; quit;
/* Do second cluster */
proc surveyreg data=&dset; cluster &cluster2; model &yvar= &xvars / covb; ods output CovB = Cov2 ParameterEstimates = params; quit;

/* Now get the covariances numbers created above. Calc coefs, SEs, t-stats, p-vals using COV = COV1 + COV2 - COVI*/
proc iml; reset noprint; use params;
read all var{Parameter} into varnames;
read all var _all_ into b;
use Cov1; read all var _num_ into x1;
use Cov2; read all var _num_ into x2;
use CovI; read all var _num_ into x3;

cov = x1 + x2 - x3; /* Calculate covariance matrix */
dfe = b[1,3]; stdb = sqrt(vecdiag(cov)); beta = b[,1]; t = beta/stdb; prob = 1-probf(t#t,1,dfe); /* Calc stats */

print,"Parameter estimates",,varnames beta[format=8.4] stdb[format=8.4] t[format=8.4] prob[format=8.4];

conc = beta || stdb || t || prob;
cname = {"estimates" "stderror" "tstat" "pvalue"};
create clus2dstats from conc [ colname=cname ];
append from conc;

conc = varnames;
cname = {"varnames"};
create names from conc [ colname=cname ];
append from conc;
quit;

data clus2dstats; merge names clus2dstats; run;
%MEND clus2OLS;

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Shmuel — Thu, 16 Feb 2017 20:05:51 GMT

Try running with system option: -memsize = 0;

That will enable use memory and virtual memory as much as needed.

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

SASKiwi — Thu, 16 Feb 2017 20:06:51 GMT

Run your program with the SAS option FULLSTIMER and the SAS log will report how much memory is being used in each step.

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

starky987 — Thu, 16 Feb 2017 20:11:18 GMT

How do I set a system option?

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

starky987 — Thu, 16 Feb 2017 20:49:24 GMT

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

starky987 — Thu, 16 Feb 2017 20:27:07 GMT

I received the following information from fulltimer

NOTE: PROCEDURE SURVEYREG used (Total process time):
real time 21.64 seconds
user cpu time 19.18 seconds
system cpu time 2.48 seconds
memory 572712.68k
OS Memory 652840.00k
Timestamp 02/16/2017 03:26:20 PM
Step Count 73 Switch Count 0

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

cau83 — Thu, 16 Feb 2017 20:47:24 GMT

It doesn't help you solve this with PROC SURVEYREG, but I've had a similar issue in the past and was not able to resolve through the suggested methods. I think they work in the right environment. I'm using a server-delivered SAS and therefore do not have complete control over how much memory (or at least, that's my working theory).

I've ended up using a different approach: Create a random number as a variable in that data set, and then do a selection off that. So for instance, create a random number from 0 to 1, then only output when the value is less than 0.05 (5% sample).

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Shmuel — Thu, 16 Feb 2017 21:07:00 GMT

@starky987 wrote:

How do I set a system option?

Find the -memsize option in the config file and change the value to zero

or in your sas execution:

"C:\Program Files\SASHome\SASFoundation\9.4\sas.exe"

-CONFIG "C:\Program Files\SASHome\SASFoundation\9.4\nls\en\sasv9.cfg" -MEMSIZE 0

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

starky987 — Thu, 16 Feb 2017 21:08:58 GMT

I went ahead and made that change with no luck. I've posted the fullstimer results and I think (not sure though) it is an issue related to the OS memory from that log?

I'm not entirely sure, I just know its not possible that this file takes up more than the 32G of ram available.

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Kurt_Bremser — Fri, 17 Feb 2017 10:37:55 GMT

According to the fullstimer log, you had less that 1GB memory in use. Could it be that there is a system setting (like the UNIX ulimit) in place that limits the resources available for your SAS process?

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

starky987 — Fri, 17 Feb 2017 10:51:15 GMT

I am afraid that may be the case Kurt but I don't know why or how to fix it. I am running Windows 10 Pro on a personal machine and am the only user. Any thoughts?

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Kurt_Bremser — Fri, 17 Feb 2017 11:16:54 GMT

I have to admit that my Windows admin skills are virtually nonexistent (compared to my AIX admin skills). Even googling did not provide me with a hint.