Proc Freq - Error: Table too large to process

[HyunJee]

I am trying to create an output dataset of the following variable comparisons using proc freq. I am able to do this with one subset of my population which is small, but when I run the same coding on the larger dataset, the error that the table is too large to process appears in the log. Is there a way to fix this in a setting or do I need to try and create the variable comparisons in a different program? Thank you for any help you can give.

proc freq data=hpv_single_reasons;

tables r_HPV_recode_1*r_HPV_recode_2*r_HPV_recode_3*r_HPV_recode_4*r_HPV_recode_5*r_HPV_recode_6*r_HPV_recode_7*r_HPV_recode_8*

r_HPV_recode_9*r_HPV_recode_10*r_HPV_recode_11*r_HPV_recode_12*r_HPV_recode_13*r_HPV_recode_14*r_HPV_recode_15*r_HPV_recode_16*

r_HPV_recode_17*r_HPV_recode_18*r_HPV_recode_19*r_HPV_recode_20*r_HPV_recode_21*r_HPV_recode_22*r_HPV_recode_23*r_HPV_recode_24*

r_HPV_recode_25*r_HPV_recode_26*r_HPV_recode_27/noprint out = hpv_single_reasons;

run;

[Alpay · Accepted solution]

I think Proc Freq might be trying to utilize memory to fit all the combinations of class variables in it.

Have you tried Proc Summary instead? It will provide you the counts not the percentages which can be computed easily once you get the counts.

proc summary data=hpv_single_reasons nway missing;

    class  r_HPV_recode_1 r_HPV_recode_2 r_HPV_recode_3 r_HPV_recode_4 r_HPV_recode_5 r_HPV_recode_6

r_HPV_recode_7 r_HPV_recode_8 r_HPV_recode_9 r_HPV_recode_10 r_HPV_recode_11 r_HPV_recode_12 r_HPV_recode_13

r_HPV_recode_14 r_HPV_recode_15 r_HPV_recode_16 r_HPV_recode_17 r_HPV_recode_18 r_HPV_recode_19

r_HPV_recode_20 r_HPV_recode_21 r_HPV_recode_22 r_HPV_recode_23 r_HPV_recode_24

r_HPV_recode_25 r_HPV_recode_26 r_HPV_recode_27;

output out=hpv_single_reasons(rename=_freq_=Count drop=_type_);

run;

[Astounding]

When the data requires too much memory to process, one approach to work around it is to use extra CPU time instead of memory.  In the extreme, you could sort your data by all 27 variables, and then use a DATA step with a BY statement to count observations.

Alternatively, if by some chance your data are already in order by one of the variables, you could use that as a BY variable in PROC FREQ (or PROC SUMMARY for that matter).  For example, if your data were in order by r_HPV_recode_1, but not in order by any of the other variables, you could use:

proc freq data=hpv_single_reasons;

  by r_HPV_recode_1;

  tables (same list, but remove r_HPV_recode_1) / out=hpv_single_reasons;

run;

The counts will be the same as if your original program had worked, although the percentages will be different.

There's no guarantee it will work, but it certainly has a good chance.  You might have to switch 2 or 3 variables to the BY statement instead of just 1, in which case the SORT + DATA step might be preferable.

Good luck.