topic Re: ERROR: The SAS System stopped processing this step because of insufficient memory. in New SAS User

ERROR: The SAS System stopped processing this step because of insufficient memory.

SBS — Sat, 06 Oct 2018 15:29:22 GMT

I am using SAS University Edition on Linux with Oracle VirtualBox. The base memory I have allocated is 24GB. Yet when I run the following code I get "ERROR: The SAS System stopped processing this step because of insufficient memory.":

proc glimmix data=MY.Data1 method=quad;
class person_gender person_id;
model NumberOfPronounsUsed = person_gender / link=log s dist=negbinomial offset=log_TotalWordsInSentence;
random int / subject=person_id;
run;

Please note that number of observations in MY.Data1 is 3820025 (i.e., 3.8 million). Is there no way that SAS University Edition can handle this? Please help!

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Kurt_Bremser — Sat, 06 Oct 2018 17:01:07 GMT

SAS UE is a learning tool with limited capacity, in terms of internal storage and memory. Reduce your data.

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

SBS — Sat, 06 Oct 2018 15:48:40 GMT

Thank you for quick response. Is there any free/paid version on Linux that can handle this data size?

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

SBS — Sat, 06 Oct 2018 16:06:01 GMT

Also why is that SAS UE can process the following mixed model in just a minute, but not the negative binomial (described earlier) for same size of data (3.8 million)?

proc mixed data=MY.Data1 method=reml covtest;
class person_gender person_id;
model PercetageOfPronoungs=person_gender /solution;
random intercept / subject=person_id; 
run;

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Tom — Sat, 06 Oct 2018 18:22:26 GMT

The amount of memory needed to process a model is not really related to the size of the data. It is related to the size of the working memory that SAS needs to process the model. The negative binomial must use more memory to process the model.

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

SBS — Sat, 06 Oct 2018 18:24:53 GMT

Thanks Tom. Is this a limitation of SAS implementation in the UE edition alone?

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Tom — Sat, 06 Oct 2018 18:28:07 GMT

The UE version is purposely limited in the amount of memory it can use. If you want to use SAS for real then buy a copy.

But note that any version of SAS (or any other analytic software) will have limits on the size of problems it can solve. Computers are finite machines.

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

SBS — Sat, 06 Oct 2018 18:29:21 GMT

Thanks Tom. I guess I will have to try with buying a copy.

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

SBS — Sat, 06 Oct 2018 22:13:52 GMT

I spent considerable time just to buy a license for SAS. I just can't find any!!

BTW, can one use HPMIXED or HPGLIMMIX in SAS UE to deal with large data?

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Ksharp — Sun, 07 Oct 2018 10:44:35 GMT

Your data is too large for HPMIXED or HPGLIMMIX .

Try PROC GEE or PROC GENMOD for GEE model .

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

SBS — Sun, 07 Oct 2018 13:55:56 GMT

Thanks Ksharp. Does either of these PROCs support negative binomial with mixed effects?

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

SBS — Sun, 07 Oct 2018 22:06:26 GMT

Would this be correct syntax?

proc genmod data=MY.Data1;
    class person_ender person_id;
    model NumberOfPronounsUsed = person_gender  / type3 dist=negbin offset=log_TotalWordsInSentence;
    Repeated subject=person_id/sorted type=exch;

Please advise!

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Ksharp — Mon, 08 Oct 2018 12:19:43 GMT

Yes. But these PROCs support negative binomial for GEE model(very like mixed model) ,not for mixed effects.

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

SBS — Mon, 08 Oct 2018 12:45:32 GMT

Sorry, I am not aware of conceptual difference. For the particular problem that I am looking at here, is the genmod incorrect? Please help!

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Ksharp — Mon, 08 Oct 2018 13:05:33 GMT

genmod is correct .

But you gotta know proc genmod (total population effect) is different from proc mixed (margin effect)

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

Ksharp — Mon, 08 Oct 2018 13:07:26 GMT

genmod is correct .

But you gotta know proc genmod (total population effect) is different from proc mixed (margin effect)

Re: ERROR: The SAS System stopped processing this step because of insufficient memory.

SBS — Mon, 08 Oct 2018 13:14:32 GMT

Thank you so much Ksharp.