topic insufficient memory with cluster in surveyreg in SAS Programming

insufficient memory with cluster in surveyreg

margalits1 — Tue, 01 Feb 2022 15:10:49 GMT

Hello, I am running the following regression for a sample of 104,000 observations (some are omitted because of missing values).

I want to add clusters by firm (there are approx.1700 firms in the sample ). I already tried to increase the memsize manually to 6G, but with no luck.

I tried both locally and via WRDS servers.

This is the code:

%macro reg_s(var_name, data_name);
proc surveyreg data=&data_name;
cluster gvkey;
CLASS gvkey year_data;
MODEL &var_name=incu_post_ipo sign_inc_p sign_p sale_rank market_share ind_code_count gvkey year_data /SOLUTION;
ods output ParameterEstimates=&var_name._&data_name;
RUN; 
ods trace off;

%mend;


%reg_s(capx_1,ff);
run;

This is the error output:

NOTE: In data set FF, total 104383 observations read, 43366 observations with missing values are
      omitted.
ERROR: The SAS System stopped processing this step because of insufficient memory.
NOTE: PROCEDURE SURVEYREG used (Total process time):
      real time           1.06 seconds
      user cpu time       0.89 seconds
      system cpu time     0.10 seconds
      memory              51160.06k
      OS Memory           77100.00k
      Timestamp           01/02/2022 04:54:23 àçäØö
      Step Count                        20  Switch Count  0

2452  run;

P.S when I run the code without the clusters this is what I get:

NOTE: In data set FF, total 104383 observations read, 43366 observations with missing values are
      omitted.
NOTE: The data set WORK.CAPX_1_FF has 1127 observations and 6 variables.
NOTE: PROCEDURE SURVEYREG used (Total process time):
      real time           42.59 seconds
      user cpu time       39.06 seconds
      system cpu time     0.87 seconds
      memory              101172.81k
      OS Memory           126420.00k
      Timestamp           01/02/2022 05:06:44 àçäØö
      Step Count                        22  Switch Count  819

I already read some relevant Q&A but couldn't find an answer to my case.

Thank you in advance.

Re: insufficient memory with cluster in surveyreg

Rick_SAS — Tue, 01 Feb 2022 15:54:30 GMT

I am not an expert in survey regression, but are you sure you want to include gvkey (the clustering variable) on the MODEL statement? I noticed in the documentation examples, the variable on the CLUSTER statement is not included on the MODEL statement.

Re: insufficient memory with cluster in surveyreg

margalits1 — Wed, 02 Feb 2022 07:09:51 GMT

I want to have both fixed effects and clustering for gvkey (the firm).
Perhaps I should use a different model then?

Re: insufficient memory with cluster in surveyreg

andreas_lds — Wed, 02 Feb 2022 12:21:00 GMT

Please post the result of

proc options group=memory;
run;

and some information about the hardware used: how much memory is installed, how much is free?

It seems strange that the first run terminates with ~50% memory used, compared with the second run.

Re: insufficient memory with cluster in surveyreg

margalits1 — Thu, 03 Feb 2022 09:51:55 GMT

Thanks for your reply,
I actually read a bit more and found that indeed surveyreg has memory issues with clusters, and now use GENMOD instead, which captures both f.e and two-way cluster, just as I wanted.
Thanks again