Insufficient Memory ... proc SURVEYREG

[Rakkoo]

Dear folks,

I couldn't find the solution in the previous postings, so let me post my question here. I'd like to run the following model

proc surveyreg data=timediff;
strata _ststr;
cluster _psu;
model timediff = white Asian NHPI AIAN OthRace Multi Hispani Male age18_24 age30_34 age35_39 age40_44 age45_49 age50_54 age55_59 age60_64 age65_69 age70_74 age75_79 age80_XX empy selfempy Hommake Student Retire Unable Divorced Widowed Separated Nevermarried Unmacouple never_atten G1_8 G9_11 edu_HS Edu_SomeC In10_15 In15_20 In20_25 In25_35 In35_50 In50_75 In75_xx Past1_2 Past2_5 Past5_xx Past_never PerDoc1_xx PerDoc_None HLTHPLN1 MEDCOST HLTH_VeryGood HLTH_Good HLTH_Fair HLTH_Poor;
weight weight;
run;

Unfortunately, I've run into this error:

ERROR: The SAS System stopped processing this step because of insufficient memory.

However, I'm not sure if the problem stems from the physical memory size, given these two considerations:

First, I calculated the required memory size (based on this link: http://support.sas.com/documentation/cdl/en/statug/63962/HTML/default/viewer.htm#statug_surveyreg_a0...) and found that less than 600MB is required.

• # observations (unweighted) = 886,167
• # strata = 1,284
• # cluster = 45,804
• # parameters = 56

Second, my computer has much larger memory:

• Windows 7 Professional (Service Pack 1)
• System type: 64-bit OS
• RAM 8GB

Let me show you the memory setting below:

SAS (r) Proprietary Software Release 9.4 TS1M3

Group=MEMORY
SORTSIZE=MAX Specifies the amount of memory that is available to the SORT procedure.
SUMSIZE=0 Specifies a limit on the amount of memory that is available for data summarization procedures when class
variables are active.
MAXMEMQUERY=0 Specifies the maximum amount of memory that is allocated for procedures.
MEMBLKSZ=16777216 Specifies the memory block size for Windows memory-based libraries.
MEMMAXSZ=2147483648
Specifies the maximum amount of memory to allocate for using memory-based libraries.
LOADMEMSIZE=0 Specifies a suggested amount of memory that is needed for executable programs loaded by SAS.
MEMSIZE=6442450944
Specifies the limit on the amount of virtual memory that can be used during a SAS session.
REALMEMSIZE=0 Specifies the amount of real memory SAS can expect to allocate.

I am not sure if I have not configured it appropriately, or if I have to install more physical memory, or if the solution is something else. Would you please help?

Thanks a lot!

Rakkoo

[ballardw]

Are you by any chance working with BRFSS data? The strata and cluster variable names look very familiar as well as the apparent categories for a bunch of demographic variables. I have to assume that most of those are coded as dichotomous variables as member/not member of population. The numbers of observations and strata makes me suspect this is an entire US data set.

If so, I am curious what the Timedif variable represents.

[Rakkoo]

Yes, I'm working on the BRFSS data of 2011 and 2012. Yes, it is the entire US data. Yes, all covariates are dummy variables, even though I'm considering to include some of them as ordinal variables and reduce the number of parameters. [timedif] is a calculated variable: the length of time since the last HIV test.

[SASKiwi · Accepted solution]

Try running your program using the SAS option FULLSTIMER. This should report the actual memory stats for SURVEYREG in the log. 

[Rakkoo]

Thank you. Let me try that option. BTW, I've found that the number of clusters is much larger, 367,222, which necessitates over 4GB memory, according to the formula. Then, I've found that SAS sets max 2GB memory by default. I've edited the sasv9.cfg file to increase the memory, but still find the insufficient memory error.

[Rakkoo]

Thanks. The FULLSTIMER option was very useful to identify the problem. I found two: (1) I initially needed larger physical memory for the large number of clusters; and (2) I did not even have to worry about insufficient memory after all, because I did not have to consider clusters in my model. I have learned that BRFSS sampling has not used the cluster component since 2011.

[ezequias]

I am facing this error on SAS Studio. It is not possible to change settings on SAS Studio I guess.

Could you give some advice? Is this really a limitation?

Sincerely

Ezequias Rocha

[rcrs2]

Due to the insufficient memory error, SAS was not able to get me the SURVEYREG results. PROC SURVEYREG needs a large amount of memory and takes a long time to calculate, as it creates a covariance matrix for each cluster. It luckily turned out that I did not need to specify cluster for my model. So I didn't have to solve the problem. Also, I've never used SAS Studio. Sorry I cannot give you much help. 

[ballardw]

---

@Rakkoo wrote:

Yes, I'm working on the BRFSS data of 2011 and 2012. Yes, it is the entire US data. Yes, all covariates are dummy variables, even though I'm considering to include some of them as ordinal variables and reduce the number of parameters. [timedif] is a calculated variable: the length of time since the last HIV test.

---

Since your Timediff variable is somewhat dependent on age I would think that having age as continous variable would be better than the agegroup variables. For example NO 18 year old could have a Timediff of 20. Make sure that your refused or don't know ages are missing if you go this route. This may also have an impact on your memory issue.

I would also expect the Timediff to be missing for refused and don't know for the date of last test.

You may also want to switch to Surveylogistic if you have an interest in is the time greater than or less than some key value (5 years or such).

BTW I believe that you may want to use the NOMCAR option as data missing due to skip patterns in the survey is not missing at random.

[Rakkoo]

Thank you for great suggestions. I'll include NOMCAR option in the command. Age is potentially to be included as one continuous variable, but is currently included as a bunch of dummy variables for the linearity check purpose. Yes, I've handled those missing data. I've also used proc surveylogistic, as there are many studies using logistic regression. However, I'm trying to go further and regress on the continuous variable.

[ballardw]

Please look at this example of regression using some dummy data and what may happen with a bunch of dichotomous variables.

data example;
   do x= 1 to 1000;
      y= x + ranuni(123);
      d1= (x<300);
      d2= (300 le x <600);
      d3= (x ge 600);
      output;
   end;
run;

proc reg data=example;
   var x y d:;
   model y = x;
   run;
   model y = x d:;
   run;
quit; 

[Rakkoo]

I'm afraid that I don't get your point. Would you please explain for me? 🙂

When I ran your code, y is perfectly predicted by x only (R-sq=1.0000).

[ballardw]

The output for the model y = x d: ; Statement is what to look at. The three D variables correspond to splitting age into 3 dichotomous varialbles. Note that you get 0 for a parameter for the 3d and some warning notes in the output.

Surveyreg will have similar issues with the dichotomous variables that have no overlaps as the "last" one listed in the model is completely dependent on the values of the others in the group.

[Rakkoo]

You're right. So, I included only two of the three d# variables so that the omitted dummy variable can serve as reference. Likewise, in my program, there is one omitted dummy variable (reference) in every set of dummy variables. 

[Theo_Gh]

Hi Rakko,
May I know how exactly you fixed this issue because I'm having the same issue and can't really figure the solutions provided here.

Thanks,
Theo