topic Re: Overfitting in logistic regression! in New SAS User

Overfitting in logistic regression!

joanatomeribeir — Mon, 08 Jun 2020 14:41:56 GMT

Hello,

I'm a new sas user, so firstly I'm sorry if these questions are dumb. 🙂

I'm basically doing a binary logistic regression, in order to predict my target variable (inactive=0, active=1) and I've randomly split the data into training (70%) and testing data (30%).

I used the proc logistic to run the logistic regression and now I need to understand if my model is overfitting /underfitting the data or not.

Does anyone have any suggestions for analyzing overfitting with proc logistic? Are we able to do learning curves?

proc logistic data=train;

class country gender / param=glm;

model y(event='1')=income var2 var3 /link=logit ctable

selection=backward slstay=0.05 hierarchy=single technique=fisher outroc=troc maxiter=50;

score data=test out=valpred outroc=vroc;

roc; roccontrast;

run;

Thank you all in advance,

Joana

Re: Overfitting in logistic regression!

Reeza — Mon, 08 Jun 2020 14:48:50 GMT

ROC and ROC contrast are the curves usually used, is that what you mean by learning curves?

You can also look at PROC PLM.

Re: Overfitting in logistic regression!

PaigeMiller — Mon, 08 Jun 2020 15:19:19 GMT

You can compare the training and validation data sets using PROC LOGISTIC

http://support.sas.com/kb/39/724.html

Re: Overfitting in logistic regression!

joanatomeribeir — Mon, 08 Jun 2020 15:21:18 GMT

By learning curves, I meant to plot the loss of the train and test over time to understand if the model is overfitted or not.

Re: Overfitting in logistic regression!

joanatomeribeir — Mon, 08 Jun 2020 16:52:08 GMT

I did it, i scored the data with score statement on proc logistic, but I want to understand if the model is overfitted or not..

Re: Overfitting in logistic regression!

PaigeMiller — Mon, 08 Jun 2020 17:54:59 GMT

@joanatomeribeir wrote:
By learning curves, I meant to plot the loss of the train and test over time to understand if the model is overfitted or not.

What do you mean by "loss of the train and test over time"? The general definition of overfitting does not include a time-related component.

Re: Overfitting in logistic regression!

joanatomeribeir — Tue, 09 Jun 2020 09:02:06 GMT

Sorry, I didn't mean over time, I meant over training set size to understand if the model is overfitting/underfitting or if it is fitting the model well...

Re: Overfitting in logistic regression!

PaigeMiller — Tue, 09 Jun 2020 10:51:57 GMT

What are the two lines in these graphs? Are they the confidence intervals of the logistic regression model coefficients? Please be specific.

Re: Overfitting in logistic regression!

Ksharp — Tue, 09 Jun 2020 11:39:10 GMT

If your sas is above 9.4M4 , you could try GOF option.

model .........../gof ;

@Rick_SAS wrote a blog about it before ,and compare statistic V.S. machine learning .

If your sas version is low ,try LACKFIT option.

model ........../ lackfit firth ;

Re: Overfitting in logistic regression!

joanatomeribeir — Tue, 09 Jun 2020 11:43:50 GMT

I'm sorry, you are right! Basically its the training and the validation set and i need to compare it to understand the bias and variance between the models.

More specifically, I would like to understand if for a binary logistic regression if it makes sense to plot the log loss (y axis) and the training set size (x axis) instead of plotting MSE.

I'm quite lost of this issue....

Re: Overfitting in logistic regression!

PaigeMiller — Tue, 09 Jun 2020 12:22:11 GMT

As far as I know, there is no built-in method in PROC LOGISTIC or PROC HPLOGISTIC to do this.

Re: Overfitting in logistic regression!

Ksharp — Wed, 10 Jun 2020 12:05:31 GMT

https://blogs.sas.com/content/iml/2019/01/30/model-validation-machine-learning.html

Re: Overfitting in logistic regression!

Ksharp — Fri, 12 Jun 2020 11:32:46 GMT

You could try PROC HPGENSELECT +PARTITION as same as I refer to Rick's blog.