BookmarkSubscribeRSS Feed
starky987
Calcite | Level 5

Hello,

 

I have received the following error message:

 

ERROR: The SAS System stopped processing this step because of insufficient memory.
NOTE: PROCEDURE SURVEYREG used (Total process time):
real time 21.09 seconds
cpu time 21.15 seconds

 

To put this into context, I am trying to run proc survey reg clustering errors along two dimensions with ~5 million observations. The macro I am using for this is posted below. I am using SAS 9.4 on Windows 10 Pro. My computer has 32 GB of ram and I believe I have set my memory options appropriately...I right clicked my SAS 9.4 icon, chose properties, and ammended the "target" line to say 

 

"C:\Program Files\SASHome\SASFoundation\9.4\sas.exe" -CONFIG "C:\Program Files\SASHome\SASFoundation\9.4\nls\en\sasv9.cfg" -MEMSIZE 30G

 

Essentially just adding the part in bold.

 

The macro I used is listed below and I am trying to include 16 independent variables. I find it hard to imagine that 30GB of memory is not sufficient to complete this task.

 

Thanks!

 

%MACRO clus2OLS(yvar, xvars, cluster1, cluster2, dset);
/* do interesection cluster*/
proc surveyreg data=&dset; cluster &cluster1 &cluster2; model &yvar= &xvars / covb; ods output CovB = CovI; quit;
/* Do first cluster */
proc surveyreg data=&dset; cluster &cluster1; model &yvar= &xvars / covb; ods output CovB = Cov1; quit;
/* Do second cluster */
proc surveyreg data=&dset; cluster &cluster2; model &yvar= &xvars / covb; ods output CovB = Cov2 ParameterEstimates = params; quit;

/* Now get the covariances numbers created above. Calc coefs, SEs, t-stats, p-vals using COV = COV1 + COV2 - COVI*/
proc iml; reset noprint; use params;
read all var{Parameter} into varnames;
read all var _all_ into b;
use Cov1; read all var _num_ into x1;
use Cov2; read all var _num_ into x2;
use CovI; read all var _num_ into x3;

cov = x1 + x2 - x3; /* Calculate covariance matrix */
dfe = b[1,3]; stdb = sqrt(vecdiag(cov)); beta = b[,1]; t = beta/stdb; prob = 1-probf(t#t,1,dfe); /* Calc stats */

print,"Parameter estimates",,varnames beta[format=8.4] stdb[format=8.4] t[format=8.4] prob[format=8.4];

conc = beta || stdb || t || prob;
cname = {"estimates" "stderror" "tstat" "pvalue"};
create clus2dstats from conc [ colname=cname ];
append from conc;

conc = varnames;
cname = {"varnames"};
create names from conc [ colname=cname ];
append from conc;
quit;

data clus2dstats; merge names clus2dstats; run;
%MEND clus2OLS;

11 REPLIES 11
Shmuel
Garnet | Level 18

Try running with system option:  -memsize = 0;

That will enable use memory and virtual memory as much as needed.

Shmuel
Garnet | Level 18

@starky987 wrote:

How do I set a system option?


Find the -memsize option in the config file and change the value to zero

or in your sas execution:

 

"C:\Program Files\SASHome\SASFoundation\9.4\sas.exe" 

    -CONFIG "C:\Program Files\SASHome\SASFoundation\9.4\nls\en\sasv9.cfg" -MEMSIZE 0

 

starky987
Calcite | Level 5

I went ahead and made that change with no luck. I've posted the fullstimer results and I think (not sure though) it is an issue related to the OS memory from that log?

 

I'm not entirely sure, I just know its not possible that this file takes up more than the 32G of ram available.

Kurt_Bremser
Super User

According to the fullstimer log, you had less that 1GB memory in use. Could it be that there is a system setting (like the UNIX ulimit) in place that limits the resources available for your SAS process?

SASKiwi
PROC Star

Run your program with the SAS option FULLSTIMER and the SAS log will report how much memory is being used in each step.

starky987
Calcite | Level 5

I received the following information from fulltimer

 

NOTE: PROCEDURE SURVEYREG used (Total process time):
real time 21.64 seconds
user cpu time 19.18 seconds
system cpu time 2.48 seconds
memory 572712.68k
OS Memory 652840.00k
Timestamp 02/16/2017 03:26:20 PM
Step Count 73 Switch Count 0

cau83
Pyrite | Level 9

It doesn't help you solve this with PROC SURVEYREG, but I've had a similar issue in the past and was not able to resolve through the suggested methods. I think they work in the right environment. I'm using a server-delivered SAS and therefore do not have complete control over how much memory (or at least, that's my working theory).

 

I've ended up using a different approach: Create a random number as a variable in that data set, and then do a selection off that. So for instance, create a random number from 0 to 1, then only output when the value is less than 0.05 (5% sample). 

starky987
Calcite | Level 5
I am afraid that may be the case Kurt but I don't know why or how to fix it. I am running Windows 10 Pro on a personal machine and am the only user. Any thoughts?

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 11 replies
  • 32722 views
  • 2 likes
  • 5 in conversation