topic Re: Regularized regression in Statistical Procedures

Regularized regression

DNYabs — Mon, 30 Apr 2018 15:54:08 GMT

I am running a regularized regression on several traits using the following code:

Proc glmselect data = DalReg1 plots(stepaxis=normb)=coefficients;
Model TW = Protein TGW SGD GL GW Size Shape / selection = LASSO(stop=none choose = cvex);
run;

The output is great. However, I am wondering how to obtain standard errors for each coefficient. Help suggestions on this, please?

Thanks,

Dalitso

Re: Regularized regression

ballardw — Mon, 30 Apr 2018 16:02:08 GMT

The standard error for the coefficients appears in the parameter estimates results which should be in the output by default. Do mean to ask how to get that information into a data set?

Re: Regularized regression

DNYabs — Mon, 30 Apr 2018 17:07:23 GMT

Ballardw: Here is the output. There is no SE.

SAS Output

Analysis of Variance Source DF Sum of Squares Mean Square F Value Model Error Corrected Total

5	4523.88515	904.77703	208.11
1271	5525.89163	4.34767
1276	10050

Root MSE Dependent Mean R-Square Adj R-Sq AICAI CCS BC CVEX PRESS

2.08511
58.92247
0.4501
0.4480
3161.71694
3161.80520
1913.63055
4.85730

Parameter Estimates Parameter DF Estimate Intercept Protein TGW SGD GL Shape

1	19.141464
1	-0.221838
1	0.192900
1	15.111831
1	0.040392
1	0.195827

Re: Regularized regression

StatDave — Mon, 30 Apr 2018 17:50:21 GMT

You might consider doing LASSO selection via PROC NLMIXED instead as illustrated in this note.

Re: Regularized regression

DNYabs — Mon, 30 Apr 2018 18:53:38 GMT

Hi Dave,

I am not sure if I am familiar with NLMIXED. Is there any other way with proc GLMSELECT? If not I might just to have a go at NLMIXED and see.

Thanks Dave.

Re: Regularized regression

PGStats — Mon, 30 Apr 2018 20:42:28 GMT

There are no SE provided when variable selection is performed with LASSO. There might be a good reason for that. Models resulting from variable selection methods do not account in their parameter estimates SE for model uncertainty. You can get parameter SEs for the chosen model, conditional on that choice, with other regression procedures, such as GLM, GENMOD or GLIMMIX.

Re: Regularized regression

DNYabs — Mon, 30 Apr 2018 21:52:55 GMT

Thanks PG. I agree. I thought there must be a good reason for not having SEs in LASSO procedure. I might have to do some more literature review on this. I chose LASSO because I have multicollinearity in my data but I am curious what SEs would be, if it is possible to generate them. Thanks again!

Re: Regularized regression

Ksharp — Tue, 01 May 2018 11:09:47 GMT

proc hpgenselect data=sashelp.class ;
class sex;
model weight = sex height age/ CL ; 

selection method=Lasso(choose=SBC) details=all;
performance details;
run;

You will see :

 NOTE: The CL option is not available for the LASSO method.
 NOTE: The HPGENSELECT procedure is executing in single-machine mode.
 NOTE: * Optimal Value of Criterion
 NOTE: There were 19 observations read from the data set SASHELP.CLASS.

Re: Regularized regression

DNYabs — Tue, 01 May 2018 21:32:43 GMT

Ksharp,

This is what I have seen:

NOTE: The CL option is not available for the LASSO method.
NOTE: The HPGENSELECT procedure is executing in single-machine mode.
NOTE: * Optimal Value of Criterion
NOTE: There were 1496 observations read from the data set WORK.DALREG1.
NOTE: PROCEDURE HPGENSELECT used (Total process time):
real time 1.07 seconds
cpu time 0.51 seconds

I just read about Bayesian LASSO that has the ability to generate SE. However, it requires a macro, an area I am, sadly, not competent with. Any help from anybody please?

Thanks,

DNY