topic Re: Regression: Binary Response and 100,000s of Class Levels in Statistical Procedures

Regression: Binary Response and 100,000s of Class Levels

rick_b — Sun, 02 Jun 2024 20:33:04 GMT

I am running a regression having a binary response variable and need to estimate fixed effects for hundreds of thousands of class levels (eleven_digit_account_id). Each code below produces hundreds of thousands of coefficients, one for each class level, overwhelming system resources. Is there a way to suppress class-level coefficients in any of these procs or is there another proc that can handle a binary response variable with hundreds of thousands of class levels?

proc glimmix data=summary_statistics NOCLPRINT;

class eleven_digit_account_id;

model bet_win = net_stake last_round_profit miles_fan_bet_team eleven_digit_account_id/noint solution dist=bin link=logit;

run;

quit;

proc genmod data=summary_statistics descending;

class eleven_digit_account_id;

model bet_win = net_stake last_round_profit miles_fan_bet_team eleven_digit_account_id/noint dist=bin link=logit;

run;

quit;

proc logistic data=summary_statistics;

class eleven_digit_account_id;

model bet_win (event='1') = net_stake last_round_profit miles_fan_bet_team eleven_digit_account_id/noint;

run;

quit;

Re: Regression: Binary Response and 100,000s of Class Levels

PaigeMiller — Sun, 02 Jun 2024 20:42:12 GMT

@rick_b wrote:

I am running a regression having a binary response variable and need to estimate fixed effects for hundreds of thousands of class levels (eleven_digit_account_id).

Right here I am skeptical, I don't really think this is a good thing to do. Why do you need fixed effects for each level of account_id?

Each code below produces hundreds of thousands of coefficients, one for each class level, overwhelming system resources.

Overwhelming disk space? Or overwhelming memory? Or something else? What SAS are you using anyway? Viya, Base SAS, something else?

There is a High Performance version of PROC LOGISTIC, its called PROC HPLOGISTIC. As I understand it, this speeds up the calculations by using distributed processing. As far as I know, it doesn't really overcome a limitation of system resources.

So we are back to my first question, why do you need each of the 100,000 account id to be treated individually in the model? How does this improve the model?

Re: Regression: Binary Response and 100,000s of Class Levels

rick_b — Sun, 02 Jun 2024 21:26:23 GMT

Finance referees nearly always expect fixed effects. You have probably seen many SAS Community examples where finance researchers use firm-fixed effects (typically each unique firm is identified by its gvkey). Here, eleven_digit_account_id is a unique customer ID much like gvkey is a unique company ID. In the past when studying firms I have used proc glm in combination with the absorb statement, but my understanding is that glm isn't designed to handle binary responses.

I am using SAS 9.4. Without all those unique IDs (for example if I stick to proc logistic with a strata eleven_digit_account_id statement) the regression takes about 2.5 hours to run. When eleven_digit_account_id is included in the class and model statements, my computer stops responding. It was my understanding that in proc logistic strata can be used to specify fixed effects (https://communities.sas.com/t5/Statistical-Procedures/Suitable-quot-proc-quot-for-a-model-with-Dummy-dependent-and/td-p/541784), but in smaller subsample tests I get different results when I use the strata statement vs. when I use class and place the variable in the model statement, so I question whether it is the same.

Re: Regression: Binary Response and 100,000s of Class Levels

StatDave — Sun, 02 Jun 2024 21:37:08 GMT

You can fit a fixed effects, conditional logistic model in PROC LOGISTIC by putting your account number variable in the STRATA statement. This conditions out those parameters from the likelihood so that they are not estimated. Another option is using a GEE model by specifying that variable in the SUBJECT= option in the REPEATED statement in PROC GEE. Repeated measurements on the levels of the variable are not required to use the GEE method.