Logistic regression cstatistic of Validation set is larger than Train set

[aranganayagi]

Hi, I am very new to SAS Stats and running Logistic regression.

I am looking to get answer for two questions

1. I am getting "Model Convergence status is Quasi complete Separation of data Point detected". what would be the implication of this warning and how to solve this.

2. C-statistics of validation data set is larger than C-statistics of Training set. Is this possible? 

My expectation is Training set should perform better than validation data set.  

Attached the report for your reference.

Could you please help to get answer for the 2 questions.

[PaigeMiller]

---

@aranganayagi wrote:

Hi, I am very new to SAS Stats and running Logistic regression.

I am looking to get answer for two questions

1. I am getting "Model Convergence status is Quasi complete Separation of data Point detected". what would be the implication of this warning and how to solve this.

---

https://stats.idre.ucla.edu/other/mult-pkg/faq/general/faqwhat-is-complete-or-quasi-complete-separat...

2. C-statistics of validation data set is larger than C-statistics of Training set. Is this possible? 

My expectation is Training set should perform better than validation data set.  

Yes, it is possible, if there is just random noise as the difference between training and validation, there's no reason that the training set has to perform better, randomly the model might fit the validation better. 

[Ksharp]

Question 1:

You have sparse data for category variable .

Example:

Y   RACE

1     white     

1    white

0    white

1    black

1    black 

You could see white have both 1 and 0 , but black have only 1 .

you could remove this kind of variable .

Question 2:

Yes. Anything is possible .

Since your Train and Validate data are random sample ,anything would happen .especially the size of Validate data is smaller than Train data . (Smaller size data tend to get higher C statistic)

[aranganayagi]

Thanks Page Miller and Ksharp for the reply. It is very helpful.

Again I get 2 more questions.
1.Based on ROC curve and C statistics of train and validation set, can we determine the model is performing better.

2. Is it necessary that the model should converge. ( I mean, should we fix Quasi complete separation warning). If we dont fix, what would be the implication.

[PaigeMiller]

---

@aranganayagi wrote:
Thanks Page Miller and Ksharp for the reply. It is very helpful.

Again I get 2 more questions.
1.Based on ROC curve and C statistics of train and validation set, can we determine the model is performing better.

---

Better than what?

2. Is it necessary that the model should converge. ( I mean, should we fix Quasi complete separation warning). If we dont fix, what would be the implication.

The link I provided explains what to do in the presence of quasi-complete separation.

[Ksharp]

1.Based on ROC curve and C statistics of train and validation set, can we determine the model is performing better.

I would not trust ROC or C statisitic , I prefer to Goodness Of Fit statistic like H-L test . @Rick_SAS  has written several blog about it .

2. Is it necessary that the model should converge. ( I mean, should we fix Quasi complete separation warning). If we dont fix, what would be the implication.

Yes . I thinks so . If model is not converge , the output is not trust.

Or @Rick_SAS  might have some word to say.

[aranganayagi]

Thanks @Ksharp for the reply.

Can we understand whether the model is overfitting or underfitting from Goodness of Fit statistic like H-L test. I went through the materials but couldn't figure out. Could you please help me

[Ksharp · Accepted solution]

No. GOF only can tell you if the model fit the sample data(train dataset) well or not .

If you have good GOF statistics ,it usually hint model is NOT overfit and is NOT lackfit .

If you have sas 9.4 m6 , You could try

proc logistic ....

model ........ / GOF ;

run;

if not try 

model ......../ LACKFIT ;

Another GOF is check if the model is overdisperse :

model ............/ scale=none aggregate ;

Search sas logistic 's documentation or Rick's blog you could find it .