topic Re: Logistic regression cstatistic of Validation set is larger than Train set in New SAS User

Logistic regression cstatistic of Validation set is larger than Train set

aranganayagi — Thu, 31 Oct 2019 11:23:01 GMT

Hi, I am very new to SAS Stats and running Logistic regression.

I am looking to get answer for two questions

1. I am getting "Model Convergence status is Quasi complete Separation of data Point detected". what would be the implication of this warning and how to solve this.

2. C-statistics of validation data set is larger than C-statistics of Training set. Is this possible?

My expectation is Training set should perform better than validation data set.

Attached the report for your reference.

Could you please help to get answer for the 2 questions.

Re: Logistic regression cstatistic of Validation set is larger than Train set

PaigeMiller — Thu, 31 Oct 2019 12:25:55 GMT

@aranganayagi wrote:

Hi, I am very new to SAS Stats and running Logistic regression.

I am looking to get answer for two questions

1. I am getting "Model Convergence status is Quasi complete Separation of data Point detected". what would be the implication of this warning and how to solve this.

https://stats.idre.ucla.edu/other/mult-pkg/faq/general/faqwhat-is-complete-or-quasi-complete-separation-in-logisticprobit-regression-and-how-do-we-deal-with-them/

2. C-statistics of validation data set is larger than C-statistics of Training set. Is this possible?

My expectation is Training set should perform better than validation data set.

Yes, it is possible, if there is just random noise as the difference between training and validation, there's no reason that the training set has to perform better, randomly the model might fit the validation better.

Re: Logistic regression cstatistic of Validation set is larger than Train set

Ksharp — Thu, 31 Oct 2019 12:27:38 GMT

Question 1:

You have sparse data for category variable .

Example:

Y RACE

1 white

0 white

1 black

You could see white have both 1 and 0 , but black have only 1 .

you could remove this kind of variable .

Question 2:

Yes. Anything is possible .

Since your Train and Validate data are random sample ,anything would happen .especially the size of Validate data is smaller than Train data . (Smaller size data tend to get higher C statistic)

Re: Logistic regression cstatistic of Validation set is larger than Train set

aranganayagi — Thu, 31 Oct 2019 12:35:16 GMT

Thanks Page Miller and Ksharp for the reply. It is very helpful.

Again I get 2 more questions.
1.Based on ROC curve and C statistics of train and validation set, can we determine the model is performing better.

2. Is it necessary that the model should converge. ( I mean, should we fix Quasi complete separation warning). If we dont fix, what would be the implication.

Re: Logistic regression cstatistic of Validation set is larger than Train set

PaigeMiller — Thu, 31 Oct 2019 13:36:37 GMT

@aranganayagi wrote:
Thanks Page Miller and Ksharp for the reply. It is very helpful.

Again I get 2 more questions.
1.Based on ROC curve and C statistics of train and validation set, can we determine the model is performing better.

Better than what?

2. Is it necessary that the model should converge. ( I mean, should we fix Quasi complete separation warning). If we dont fix, what would be the implication.

The link I provided explains what to do in the presence of quasi-complete separation.

Re: Logistic regression cstatistic of Validation set is larger than Train set

Ksharp — Fri, 01 Nov 2019 11:41:05 GMT

1.Based on ROC curve and C statistics of train and validation set, can we determine the model is performing better.

I would not trust ROC or C statisitic , I prefer to Goodness Of Fit statistic like H-L test . @Rick_SAS has written several blog about it .

2. Is it necessary that the model should converge. ( I mean, should we fix Quasi complete separation warning). If we dont fix, what would be the implication.

Yes . I thinks so . If model is not converge , the output is not trust.

Or @Rick_SAS might have some word to say.

Re: Logistic regression cstatistic of Validation set is larger than Train set

aranganayagi — Sat, 02 Nov 2019 15:48:32 GMT

Thanks @Ksharp for the reply.

Can we understand whether the model is overfitting or underfitting from Goodness of Fit statistic like H-L test. I went through the materials but couldn't figure out. Could you please help me

Re: Logistic regression cstatistic of Validation set is larger than Train set

Ksharp — Sun, 03 Nov 2019 11:21:22 GMT

No. GOF only can tell you if the model fit the sample data(train dataset) well or not .

If you have good GOF statistics ,it usually hint model is NOT overfit and is NOT lackfit .

If you have sas 9.4 m6 , You could try

proc logistic ....

model ........ / GOF ;

run;

if not try

model ......../ LACKFIT ;

Another GOF is check if the model is overdisperse :

model ............/ scale=none aggregate ;

Search sas logistic 's documentation or Rick's blog you could find it .