topic Re: Insufficient memory error - proc GENMOD Poisson regression in Statistical Procedures

Insufficient memory error - proc GENMOD Poisson regression

LuC3 — Tue, 30 Nov 2021 05:02:52 GMT

I submitted the following code to calculate risk ratios using Poisson regression (my binomial models did not converge). My data set has ~3million observations so I received the following: "ERROR: The SAS System stopped processing this step because of insufficient memory."

proc genmod data=library;
class inf (ref = "0") id /param=ref;
model GA = inf /dist=poisson link=log;
repeated subject=id/type=ind;
estimate 'RR GA 1 vs. 0' inf 1 0 0 0 0/exp;
estimate 'RR GA 2 vs. 0' inf 0 1 0 0 0/exp;
estimate 'RR GA 3 vs. 0' inf 0 0 1 0 0/exp;
estimate 'RR GA 4 vs. 0' inf 0 0 0 1 0/exp;
estimate 'RR GA 5 vs. 0' inf 0 0 0 0 1/exp;
run;

Additionally, I will be running an adjusted model with 9 covariates. All variables are dichotomous.

So far, I have tried setting MEMSIZE = 8GB and then to 0 and still get the insufficient memory error.

My current memory is the following (however, since setting to 0, I get an ERROR: INVALID OPTION VALUE when trying to change again):

Group=MEMORY
SORTSIZE=1073741824
Specifies the amount of memory that is available to the SORT procedure.
SUMSIZE=0 Specifies a limit on the amount of memory that is available for data summarization procedures when class variables are active.
MAXMEMQUERY=0 Specifies the maximum amount of memory that is allocated for procedures.
MEMBLKSZ=16777216 Specifies the memory block size for Windows memory-based libraries.
MEMMAXSZ=2147483648 Specifies the maximum amount of memory to allocate for using memory-based libraries.
LOADMEMSIZE=0 Specifies a suggested amount of memory that is needed for executable programs loaded by SAS.
MEMSIZE=2147483648 Specifies the limit on the amount of virtual memory that can be used during a SAS session.
REALMEMSIZE=0 Specifies the amount of real memory SAS can expect to allocate.

I'm using SAS 9.4 on a laptop with 8.00 GB RAM (7.88 GB usable).

Does anyone know how to overcome this memory issue?

Thank you!

Re: Insufficient memory error - proc GENMOD Poisson regression

sbxkoenk — Tue, 30 Nov 2021 11:44:15 GMT

Hello,

First try to run the same PROC GENMOD without ESTIMATE statements.

If the ESTIMATE statements are the reason, we can come up with an alternative (like storing the model and do the ESTIMATEs in a subsequent PROC PLM). Or even better, by switching to LSMEANS / LSMESTIMATE / SLICE statements (if possible).

If still insufficient memory try fitting the same model with HPGENSELECT.

If it is a GEE-model, try PROC GEE as well.

Let us know the result,

Koen

Re: Insufficient memory error - proc GENMOD Poisson regression

MichaelL_SAS — Tue, 30 Nov 2021 14:50:30 GMT

If the variable ID listed as the SUBJECT= effect in the REPEATED statement is a unique identifier for the 3 million-ish observations, then I suspect that is the cause of the insufficient memory error. I believe on some machines (I'm not 100% sure on the specifics) some of the memory allocations associated with the levelization of the variables on the CLASS statement will start to run up against the max allocation size around 3 million plus levels.

As per this SAS Problems Note some changes were made to PROC GEE that can increase this max number of subjects somewhat for 64 bit hosts. I'd suggest reaching out to technical support to discuss this issue with them.

Re: Insufficient memory error - proc GENMOD Poisson regression

LuC3 — Wed, 01 Dec 2021 03:30:37 GMT

Removing the ESTIMATE statements did not help, however, using HPGENSELECT did work.

As I am unfamiliar with HPGENSELECT so wanted to ensure that exponentiating the estimates will result in a risk ratio similar to the estimates provided with GENMOD.

Thanks!