topic PROC HPGENSELECT with METHOD=LASSO runs forever in Statistical Procedures

PROC HPGENSELECT with METHOD=LASSO runs forever

lichee — Fri, 26 Apr 2024 15:37:41 GMT

Hi all,

I'm trying to use PROC HPGENSELECT with METHOD=LASSO to select covariates. I have about 370,000 observations. When I tested the code with only 8 covariates, it finished running in less than twenty minutes. When I included over 300 binary covariates (&add300cov.), it ran overnight but still did not finish. Can anyone see anything to modify in the my code below? Thanks a lot!

proc hpgenselect data=population;

class female(ref='0') race_ethncty(ref='1') &add300cov_ref.;

model success(event="1") = age female race_ethncty var4 var5 var6 var7 var8 &add300cov./ dist=binary include=(age female race_ethncty var4 var5 var6 var7 var8);

selection method=lasso;

run;

Re: PROC HPGENSELECT with METHOD=LASSO runs forever

PaigeMiller — Fri, 26 Apr 2024 16:38:13 GMT

300 binary covariates most likely will take an extremely long time to fit such a model. I don't think there's any way around that. Even with an HP PROC. But there are optimization options and tolerance options for PROC HPGENSELECT, you could try those and see if anything helps (I'm guessing they would)

As far as what else to try, you could see which of the binary predictors are highly correlated with the binary response using a Chi-squared test (two way table in PROC FREQ with the CHISQ option will get you there) and just pick a few of the best binary predictors to use in the model.

Thinking out of the box, another approach is to use logistic Partial Least Squares with all 300 binary predictors. I have no doubt that even with 300 binary predictors it would finish much more quickly, and I would be surprised if it even took an hour. However, the only software to do this that I know of is in R (https://cran.r-project.org/web/packages/plsRglm/plsRglm.pdf)

Re: PROC HPGENSELECT with METHOD=LASSO runs forever

ballardw — Fri, 26 Apr 2024 20:27:36 GMT

By any chance did you turn on any system performance monitor while that code was running?

I suspect you might see a lot of disk activity and possibly very high memory usage. When there are lots of things involved with calculations you might find that SAS is spending more time writing/reading data to and from temporary storage.