Insufficient Memory ... proc SURVEYREG

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 9
Accepted Solution

Insufficient Memory ... proc SURVEYREG

Dear folks,

I couldn't find the solution in the previous postings, so let me post my question here. I'd like to run the following model

 

proc surveyreg data=timediff;
strata _ststr;
cluster _psu;
model timediff = white Asian NHPI AIAN OthRace Multi Hispani Male age18_24 age30_34 age35_39 age40_44 age45_49 age50_54 age55_59 age60_64 age65_69 age70_74 age75_79 age80_XX empy selfempy Hommake Student Retire Unable Divorced Widowed Separated Nevermarried Unmacouple never_atten G1_8 G9_11 edu_HS Edu_SomeC In10_15 In15_20 In20_25 In25_35 In35_50 In50_75 In75_xx Past1_2 Past2_5 Past5_xx Past_never PerDoc1_xx PerDoc_None HLTHPLN1 MEDCOST HLTH_VeryGood HLTH_Good HLTH_Fair HLTH_Poor;
weight weight;
run;

 

Unfortunately, I've run into this error:

 

ERROR: The SAS System stopped processing this step because of insufficient memory.

 

However, I'm not sure if the problem stems from the physical memory size, given these two considerations:

 

First, I calculated the required memory size (based on this link: http://support.sas.com/documentation/cdl/en/statug/63962/HTML/default/viewer.htm#statug_surveyreg_a0...) and found that less than 600MB is required.

  • # observations (unweighted) = 886,167
  • # strata = 1,284
  • # cluster = 45,804
  • # parameters = 56

Second, my computer has much larger memory:

  • Windows 7 Professional (Service Pack 1)
  • System type: 64-bit OS
  • RAM 8GB

 

Let me show you the memory setting below:

 

SAS (r) Proprietary Software Release 9.4 TS1M3

Group=MEMORY
SORTSIZE=MAX Specifies the amount of memory that is available to the SORT procedure.
SUMSIZE=0 Specifies a limit on the amount of memory that is available for data summarization procedures when class
variables are active.
MAXMEMQUERY=0 Specifies the maximum amount of memory that is allocated for procedures.
MEMBLKSZ=16777216 Specifies the memory block size for Windows memory-based libraries.
MEMMAXSZ=2147483648
Specifies the maximum amount of memory to allocate for using memory-based libraries.
LOADMEMSIZE=0 Specifies a suggested amount of memory that is needed for executable programs loaded by SAS.
MEMSIZE=6442450944
Specifies the limit on the amount of virtual memory that can be used during a SAS session.
REALMEMSIZE=0 Specifies the amount of real memory SAS can expect to allocate.

 

 

I am not sure if I have not configured it appropriately, or if I have to install more physical memory, or if the solution is something else. Would you please help?

 

Thanks a lot!

Rakkoo

 


Accepted Solutions
Solution
‎09-13-2016 04:19 PM
Super User
Posts: 3,252

Re: Insufficient Memory ... proc SURVEYREG

Try running your program using the SAS option FULLSTIMER. This should report the actual memory stats for SURVEYREG in the log. 

View solution in original post


All Replies
Super User
Posts: 11,343

Re: Insufficient Memory ... proc SURVEYREG

Are you by any chance working with BRFSS data? The strata and cluster variable names look very familiar as well as the apparent categories for a bunch of demographic variables. I have to assume that most of those are coded as dichotomous variables as member/not member of population. The numbers of observations and strata makes me suspect this is an entire US data set.

 

If so, I am curious what the Timedif variable represents.

 

 

Occasional Contributor
Posts: 9

Re: Insufficient Memory ... proc SURVEYREG

Yes, I'm working on the BRFSS data of 2011 and 2012. Yes, it is the entire US data. Yes, all covariates are dummy variables, even though I'm considering to include some of them as ordinal variables and reduce the number of parameters. [timedif] is a calculated variable: the length of time since the last HIV test.

Solution
‎09-13-2016 04:19 PM
Super User
Posts: 3,252

Re: Insufficient Memory ... proc SURVEYREG

Try running your program using the SAS option FULLSTIMER. This should report the actual memory stats for SURVEYREG in the log. 

Occasional Contributor
Posts: 9

Re: Insufficient Memory ... proc SURVEYREG

Thank you. Let me try that option. BTW, I've found that the number of clusters is much larger, 367,222, which necessitates over 4GB memory, according to the formula. Then, I've found that SAS sets max 2GB memory by default. I've edited the sasv9.cfg file to increase the memory, but still find the insufficient memory error.

Occasional Contributor
Posts: 9

Re: Insufficient Memory ... proc SURVEYREG

Thanks. The FULLSTIMER option was very useful to identify the problem. I found two: (1) I initially needed larger physical memory for the large number of clusters; and (2) I did not even have to worry about insufficient memory after all, because I did not have to consider clusters in my model. I have learned that BRFSS sampling has not used the cluster component since 2011.

Super User
Posts: 11,343

Re: Insufficient Memory ... proc SURVEYREG


Rakkoo wrote:

Yes, I'm working on the BRFSS data of 2011 and 2012. Yes, it is the entire US data. Yes, all covariates are dummy variables, even though I'm considering to include some of them as ordinal variables and reduce the number of parameters. [timedif] is a calculated variable: the length of time since the last HIV test.


Since your Timediff variable is somewhat dependent on age I would think that having age as continous variable would be better than the agegroup variables. For example NO 18 year old could have a Timediff of 20. Make sure that your refused or don't know ages are missing if you go this route. This may also have an impact on your memory issue.

 

I would also expect the Timediff to be missing for refused and don't know for the date of last test.

 

You may also want to switch to Surveylogistic if you have an interest in is the time greater than or less than some key value (5 years or such).

 

BTW I believe that you may want to use the NOMCAR option as data missing due to skip patterns in the survey is not missing at random.

Occasional Contributor
Posts: 9

Re: Insufficient Memory ... proc SURVEYREG

Thank you for great suggestions. I'll include NOMCAR option in the command. Age is potentially to be included as one continuous variable, but is currently included as a bunch of dummy variables for the linearity check purpose. Yes, I've handled those missing data. I've also used proc surveylogistic, as there are many studies using logistic regression. However, I'm trying to go further and regress on the continuous variable.

 

Super User
Posts: 11,343

Re: Insufficient Memory ... proc SURVEYREG

Please look at this example of regression using some dummy data and what may happen with a bunch of dichotomous variables.

 

data example;
   do x= 1 to 1000;
      y= x + ranuni(123);
      d1= (x<300);
      d2= (300 le x <600);
      d3= (x ge 600);
      output;
   end;
run;

proc reg data=example;
   var x y d:;
   model y = x;
   run;
   model y = x d:;
   run;
quit; 
Occasional Contributor
Posts: 9

Re: Insufficient Memory ... proc SURVEYREG

[ Edited ]

I'm afraid that I don't get your point. Would you please explain for me? Smiley Happy

 

When I ran your code, y is perfectly predicted by x only (R-sq=1.0000).

Super User
Posts: 11,343

Re: Insufficient Memory ... proc SURVEYREG

The output for the model y = x d: ; Statement is what to look at. The three D variables correspond to splitting age into 3 dichotomous varialbles. Note that you get 0 for a parameter for the 3d and some warning notes in the output.

 

Surveyreg will have similar issues with the dichotomous variables that have no overlaps as the "last" one listed in the model is completely dependent on the values of the others in the group.

Occasional Contributor
Posts: 9

Re: Insufficient Memory ... proc SURVEYREG

You're right. So, I included only two of the three d# variables so that the omitted dummy variable can serve as reference. Likewise, in my program, there is one omitted dummy variable (reference) in every set of dummy variables. 

Contributor
Posts: 50

Re: Insufficient Memory ... proc SURVEYREG

Hi Rakko,
May I know how exactly you fixed this issue because I'm having the same issue and can't really figure the solutions provided here.

Thanks,
Theo
Occasional Contributor
Posts: 9

Re: Insufficient Memory ... proc SURVEYREG

Hi The,

It eventually turned out that I did not need that much memory, because the line "cluster _psu;" was to be removed. Plus, I got a funding to purchase a laptop with large RAM and fast CPU. Since then, I haven't encountered many problems...

 

BTW, do you also use the BRFSS data? If so, you may not really need cluster, because each cluster includes only one record, as far as I have checked.

 

Hope this help.

Rakkoo

☑ This topic is solved.

Need further help from the community? Please ask a new question.

Discussion stats
  • 13 replies
  • 1077 views
  • 0 likes
  • 5 in conversation