topic Re: Insufficient Memory ... proc SURVEYREG in SAS Programming

Insufficient Memory ... proc SURVEYREG

Rakkoo — Thu, 08 Sep 2016 18:34:32 GMT

Dear folks,

I couldn't find the solution in the previous postings, so let me post my question here. I'd like to run the following model

proc surveyreg data=timediff;
strata _ststr;
cluster _psu;
model timediff = white Asian NHPI AIAN OthRace Multi Hispani Male age18_24 age30_34 age35_39 age40_44 age45_49 age50_54 age55_59 age60_64 age65_69 age70_74 age75_79 age80_XX empy selfempy Hommake Student Retire Unable Divorced Widowed Separated Nevermarried Unmacouple never_atten G1_8 G9_11 edu_HS Edu_SomeC In10_15 In15_20 In20_25 In25_35 In35_50 In50_75 In75_xx Past1_2 Past2_5 Past5_xx Past_never PerDoc1_xx PerDoc_None HLTHPLN1 MEDCOST HLTH_VeryGood HLTH_Good HLTH_Fair HLTH_Poor;
weight weight;
run;

Unfortunately, I've run into this error:

ERROR: The SAS System stopped processing this step because of insufficient memory.

However, I'm not sure if the problem stems from the physical memory size, given these two considerations:

First, I calculated the required memory size (based on this link: http://support.sas.com/documentation/cdl/en/statug/63962/HTML/default/viewer.htm#statug_surveyreg_a0000000284.htm) and found that less than 600MB is required.

# observations (unweighted) = 886,167
# strata = 1,284
# cluster = 45,804
# parameters = 56

Second, my computer has much larger memory:

Windows 7 Professional (Service Pack 1)
System type: 64-bit OS
RAM 8GB

Let me show you the memory setting below:

SAS (r) Proprietary Software Release 9.4 TS1M3

Group=MEMORY
SORTSIZE=MAX Specifies the amount of memory that is available to the SORT procedure.
SUMSIZE=0 Specifies a limit on the amount of memory that is available for data summarization procedures when class
variables are active.
MAXMEMQUERY=0 Specifies the maximum amount of memory that is allocated for procedures.
MEMBLKSZ=16777216 Specifies the memory block size for Windows memory-based libraries.
MEMMAXSZ=2147483648
Specifies the maximum amount of memory to allocate for using memory-based libraries.
LOADMEMSIZE=0 Specifies a suggested amount of memory that is needed for executable programs loaded by SAS.
MEMSIZE=6442450944
Specifies the limit on the amount of virtual memory that can be used during a SAS session.
REALMEMSIZE=0 Specifies the amount of real memory SAS can expect to allocate.

I am not sure if I have not configured it appropriately, or if I have to install more physical memory, or if the solution is something else. Would you please help?

Thanks a lot!

Rakkoo

Re: Insufficient Memory ... proc SURVEYREG

ballardw — Thu, 08 Sep 2016 19:13:14 GMT

Are you by any chance working with BRFSS data? The strata and cluster variable names look very familiar as well as the apparent categories for a bunch of demographic variables. I have to assume that most of those are coded as dichotomous variables as member/not member of population. The numbers of observations and strata makes me suspect this is an entire US data set.

If so, I am curious what the Timedif variable represents.

Re: Insufficient Memory ... proc SURVEYREG

Rakkoo — Thu, 08 Sep 2016 19:56:54 GMT

Yes, I'm working on the BRFSS data of 2011 and 2012. Yes, it is the entire US data. Yes, all covariates are dummy variables, even though I'm considering to include some of them as ordinal variables and reduce the number of parameters. [timedif] is a calculated variable: the length of time since the last HIV test.

Re: Insufficient Memory ... proc SURVEYREG

SASKiwi — Thu, 08 Sep 2016 20:08:21 GMT

Try running your program using the SAS option FULLSTIMER. This should report the actual memory stats for SURVEYREG in the log.

Re: Insufficient Memory ... proc SURVEYREG

ballardw — Thu, 08 Sep 2016 20:51:46 GMT

@Rakkoo wrote:

Yes, I'm working on the BRFSS data of 2011 and 2012. Yes, it is the entire US data. Yes, all covariates are dummy variables, even though I'm considering to include some of them as ordinal variables and reduce the number of parameters. [timedif] is a calculated variable: the length of time since the last HIV test.

Since your Timediff variable is somewhat dependent on age I would think that having age as continous variable would be better than the agegroup variables. For example NO 18 year old could have a Timediff of 20. Make sure that your refused or don't know ages are missing if you go this route. This may also have an impact on your memory issue.

I would also expect the Timediff to be missing for refused and don't know for the date of last test.

You may also want to switch to Surveylogistic if you have an interest in is the time greater than or less than some key value (5 years or such).

BTW I believe that you may want to use the NOMCAR option as data missing due to skip patterns in the survey is not missing at random.

Re: Insufficient Memory ... proc SURVEYREG

Rakkoo — Fri, 09 Sep 2016 16:43:29 GMT

Thank you. Let me try that option. BTW, I've found that the number of clusters is much larger, 367,222, which necessitates over 4GB memory, according to the formula. Then, I've found that SAS sets max 2GB memory by default. I've edited the sasv9.cfg file to increase the memory, but still find the insufficient memory error.

Re: Insufficient Memory ... proc SURVEYREG

Rakkoo — Fri, 09 Sep 2016 16:53:18 GMT

Thank you for great suggestions. I'll include NOMCAR option in the command. Age is potentially to be included as one continuous variable, but is currently included as a bunch of dummy variables for the linearity check purpose. Yes, I've handled those missing data. I've also used proc surveylogistic, as there are many studies using logistic regression. However, I'm trying to go further and regress on the continuous variable.

Re: Insufficient Memory ... proc SURVEYREG

ballardw — Fri, 09 Sep 2016 19:43:21 GMT

Please look at this example of regression using some dummy data and what may happen with a bunch of dichotomous variables.

data example;
   do x= 1 to 1000;
      y= x + ranuni(123);
      d1= (x<300);
      d2= (300 le x <600);
      d3= (x ge 600);
      output;
   end;
run;

proc reg data=example;
   var x y d:;
   model y = x;
   run;
   model y = x d:;
   run;
quit;

Re: Insufficient Memory ... proc SURVEYREG

Rakkoo — Mon, 12 Sep 2016 20:18:01 GMT

I'm afraid that I don't get your point. Would you please explain for me? 🙂

When I ran your code, y is perfectly predicted by x only (R-sq=1.0000).

Re: Insufficient Memory ... proc SURVEYREG

ballardw — Mon, 12 Sep 2016 22:44:31 GMT

The output for the model y = x d: ; Statement is what to look at. The three D variables correspond to splitting age into 3 dichotomous varialbles. Note that you get 0 for a parameter for the 3d and some warning notes in the output.

Surveyreg will have similar issues with the dichotomous variables that have no overlaps as the "last" one listed in the model is completely dependent on the values of the others in the group.

Re: Insufficient Memory ... proc SURVEYREG

Rakkoo — Tue, 13 Sep 2016 20:18:32 GMT

You're right. So, I included only two of the three d# variables so that the omitted dummy variable can serve as reference. Likewise, in my program, there is one omitted dummy variable (reference) in every set of dummy variables.

Re: Insufficient Memory ... proc SURVEYREG

Rakkoo — Tue, 13 Sep 2016 20:26:48 GMT

Thanks. The FULLSTIMER option was very useful to identify the problem. I found two: (1) I initially needed larger physical memory for the large number of clusters; and (2) I did not even have to worry about insufficient memory after all, because I did not have to consider clusters in my model. I have learned that BRFSS sampling has not used the cluster component since 2011.

Re: Insufficient Memory ... proc SURVEYREG

Theo_Gh — Wed, 12 Jul 2017 17:12:35 GMT

Hi Rakko,
May I know how exactly you fixed this issue because I'm having the same issue and can't really figure the solutions provided here.

Thanks,
Theo

Re: Insufficient Memory ... proc SURVEYREG

rcrs2 — Wed, 12 Jul 2017 18:59:40 GMT

Hi The,

It eventually turned out that I did not need that much memory, because the line "cluster _psu;" was to be removed. Plus, I got a funding to purchase a laptop with large RAM and fast CPU. Since then, I haven't encountered many problems...

BTW, do you also use the BRFSS data? If so, you may not really need cluster, because each cluster includes only one record, as far as I have checked.

Hope this help.

Rakkoo

Re: Insufficient Memory ... proc SURVEYREG

ezequias — Fri, 24 Aug 2018 16:53:48 GMT

I am facing this error on SAS Studio. It is not possible to change settings on SAS Studio I guess.

Could you give some advice? Is this really a limitation?

Sincerely

Ezequias Rocha

Re: Insufficient Memory ... proc SURVEYREG

rcrs2 — Fri, 24 Aug 2018 17:27:47 GMT

Due to the insufficient memory error, SAS was not able to get me the SURVEYREG results. PROC SURVEYREG needs a large amount of memory and takes a long time to calculate, as it creates a covariance matrix for each cluster. It luckily turned out that I did not need to specify cluster for my model. So I didn't have to solve the problem. Also, I've never used SAS Studio. Sorry I cannot give you much help.

Re: Insufficient Memory ... proc SURVEYREG

ezequias — Fri, 24 Aug 2018 20:17:24 GMT

Is this topic about SAS Studio too?

I got the same message as you can evidence on the attached photo.

Regards

Ezequias.

Re: Insufficient Memory ... proc SURVEYREG

ChrisNZ — Sun, 26 Aug 2018 23:11:26 GMT

As a general rule:
1. SAS studio is a front end. The memory issue in on the server. Where is your server? Look at the memory settings there.
2. If your server is 32 bits, you are very limited in terms of accessible memory. 64 bits (both OS and SAS) allows going beyond 3GB and as high as the hardware allows.
3. You must increase REALMEMSIZE (if enough RAM is available) and MEMSIZE (at least to slightly over REALMEMSIZE, and more if you don't mind the process paging). Paging means using disk as if it were RAM, and it make the process slow to a crawl but least be able to complete.

In your case:
2. Since your log looks like a Unix log, you are probably fine
3. These settings must be changed by the system admin as they are SAS startup parameters. For a server, paging is never desirable and the proper way to configure the option is to set MEMSIZE as high as possible and REALMEMSIZE as MAX. This will affect all SAS sessions started after the change in the configuration file.

Re: Insufficient Memory ... proc SURVEYREG

ezequias — Tue, 28 Aug 2018 11:57:39 GMT

Dear @ChrisNZ I am using SAS Studio on the cloud. I have no server to perform those operations.

I didn't understand what you meant.

Sincerely

Ezequias Rocha

Re: Insufficient Memory ... proc SURVEYREG

ChrisNZ — Tue, 28 Aug 2018 21:41:35 GMT

Where your SAS processes run is your SAS server.

Whoever set up SAS for you manages your SAS environment.

That person can increase the amount of memory available to you.

If the could is managed by SAS Institute, then they would need to be contacted for this modification.

Your contract would presumably point you to the appropriate contact channels.

Does that make sense?