topic Re: ERROR: Model is too large to be fit by PROC GLIMMIX in a reasonable amount of time on this in Statistical Procedures

ERROR: Model is too large to be fit by PROC GLIMMIX in a reasonable amount of time on this

desireatem — Mon, 26 Sep 2022 15:40:00 GMT

Hello, I am fitting the glimmix procedure below but didn't get an output because the data is too big:

Proc glimmix data=female method=quad;
class age(ref='14 years or younger')grade(ref='9th grade') race4(ref='White, NH') Region1(ref='South');
model TEAMS_NO(ref='False')=year age grade race4 Region1 / ODDSRATIO SOLUTION DIST=BINARY;
RANDOM Intercept / TYPE=VC Subject=RECORD;
random intercept / subject=record weight=weight;
Title 'school predicting ';
format YEAR user. TEAMS_MU TRUEFALS. TEAMS_NO TRUEFALS. SEX SEX. age age. grade grade. race4 race. Region1 region.;
run;

496 Proc glimmix data=female method=quad;
497 class age(ref='14 years or younger')grade(ref='9th grade') race4(ref='White, NH')
497! Region1(ref='South');
498 model TEAMS_NO(ref='False')=year age grade race4 Region1 / ODDSRATIO SOLUTION DIST=BINARY;
499 RANDOM Intercept / TYPE=VC Subject=RECORD;
500 random intercept / subject=record weight=weight;
501 Title 'school predicting ';
502 format YEAR user. TEAMS_MU TRUEFALS. TEAMS_NO TRUEFALS. SEX SEX. age age. grade grade. race4
502! race. Region1 region.;
503 run;

NOTE: Some observations are not used in the analysis because of: missing fixed effects (n=11697).
NOTE: The GLIMMIX procedure is modeling the probability that TEAMS_NO='True'.
ERROR: Model is too large to be fit by PROC GLIMMIX in a reasonable amount of time on this
system. Consider changing your model.
NOTE: The SAS System stopped processing this step because of errors.
NOTE: PROCEDURE GLIMMIX used (Total process time):
real time 2.43 seconds
cpu time 2.43 seconds

Re: ERROR: Model is too large to be fit by PROC GLIMMIX in a reasonable amount of time on this

awesome_opossum — Mon, 26 Sep 2022 17:43:44 GMT

Unfortunately, it's a notorious problem with glimmix that it is a resource hog. Even with relatively small samples, it will not be able to run unless it is hosted on a supercomputer.

Barring that, your best option might be to run the model on multiple, sufficiently small samples, and hope to observe consistency. Of course, that probably wouldn't cut it for a peer-reviewed report, but may be useful for decision-making purposes based on the output.

Re: ERROR: Model is too large to be fit by PROC GLIMMIX in a reasonable amount of time on this

jiltao — Mon, 26 Sep 2022 18:36:28 GMT

First, what is the memsize? If you run the following program the log will tell you that --

proc options option=memsize value;

run;

If it is the default 2GB then you need to increase the memsize.

Secondly, you need to make sure your data is sorted by RECORD; then add ddfm=residual option in the MODEL statement.

Hope this helps,

Jill

Re: ERROR: Model is too large to be fit by PROC GLIMMIX in a reasonable amount of time on this

desireatem — Mon, 26 Sep 2022 20:11:35 GMT

THank you, I ran proc options option=memsize value; run; and below is the log file

Option Value Information For SAS Option MEMSIZE
Value: 2147483648
Scope: SAS Session
How option value set: Config File
Config file name:
C:\Program Files\SASHome\SASFoundation\9.4\nls\en\sasv9.cfg

NOTE: PROCEDURE OPTIONS used (Total process time):
real time 0.00 seconds
cpu time 0.00 seconds

Re: ERROR: Model is too large to be fit by PROC GLIMMIX in a reasonable amount of time on this

svh — Mon, 26 Sep 2022 20:23:48 GMT

I don't see "record" in the CLASS statement. Have you tried running the model with it included? Also, is there a way you can simplify to one RANDOM statement since they are both specifying a random intercept for that variable.

Re: ERROR: Model is too large to be fit by PROC GLIMMIX in a reasonable amount of time on this

jiltao — Mon, 26 Sep 2022 22:58:33 GMT

You need to increase the memsize for your SAS sessions. Here is how --

1) Right click on the notepad App and choose run as admin.

2) use file open in notepad to get to C:\Program Files\SASHome\SASFoundation\9.4\nls\en\sasv9.cfg

3) Increase memsize by modifying the following statement in the file to::

-memsize=8G

(or any other value that is within your computer capacity)

4) Save and exit.

When you re run your program, make sure that

1) your data is sorted by RECORD

2) use one RANDOM statement instead of two RANDOM statements (your two RANDOM statements are almost the same).

3) add ddfm=residual option in your MODEL statement.

Please let us know how things go.

Thanks,

Jill

Re: ERROR: Model is too large to be fit by PROC GLIMMIX in a reasonable amount of time on this

SteveDenham — Tue, 27 Sep 2022 14:01:16 GMT

In addition to the great responses from @jiltao and @svh , you might also have luck if you changed from adaptive quadrature (method=quad) to a REML-pseudolikelihood method (method=RSPL, or for marginal effects method=RMPL). This suggestion is based on the paper by Stroup and Claassen (2020) https://link.springer.com/article/10.1007/s13253-020-00402-6 (Paywall). or https://ibook.pub/pseudo-likelihood-or-quadrature-what-we-thought-we-knew-what-we-think-we-know-and-what-we-are-still-trying-to-figure.html (pdf, but you have to wade through a splash page loaded with links).

Also, when you go to a single RANDOM, be sure you use the one with the WEIGHTS option if this is based on survey data.

SteveDenham