topic Re: logistic regression: sign of estimates/coefficients in Statistical Procedures

logistic regression: sign of estimates/coefficients

adjn258 — Thu, 09 Jun 2022 20:27:30 GMT

Hi All, I have built a logistic regression model using weight of evidence transformation in SAS. My model shows good performance (in terms of discriminatory power (AUC and gini) and accuracy (Hosmer and lemeshow chi square test). However, the coefficient/estimate of one of the independent is coming out to be positive (all other coefficients, including the intercept are negative). Does this mean there is something wrong with including this factor in the model? I am being told this factor needs to be dropped because of opposite sign. Can somebody please explain if this is true? and why so?

Re: logistic regression: sign of estimates/coefficients

sbxkoenk — Fri, 10 Jun 2022 09:38:49 GMT

Question was in 'Forecasting' board.

I have moved it to 'Statistical Procedures' board as it as about Logistic Regression and WoE predictors.

Koen

Re: logistic regression: sign of estimates/coefficients

Rick_SAS — Fri, 10 Jun 2022 10:04:25 GMT

Whoever told you this is either confused or has some domain-specific knowledge that you have not shared. In general, a model can have some parameter estimates that are positive and others that are negative. It depends on the relationship between the response variable and the effect.

For example, in the PROC LOGISTIC documentation, the Getting Started example forms a model in which the Intercept term and an interaction effect have negative estimates, but the other estimates are positive. Other examples also produce models that have negative and positive parameter estimates.

Re: logistic regression: sign of estimates/coefficients

Ksharp — Fri, 10 Jun 2022 12:08:53 GMT

I totally agree with @Rick_SAS 's opinion. I think your model fit very well .

"owever, the coefficient/estimate of one of the independent is coming out to be positive "

is due to your data , NOT from Logistic Model .

If you delete that variable , you will find another variable is positive , if you delete another variable ,you will find another another variable is positive . It is endless . I think your BOSS is special to business , idiot to statistical model .

P.S. maybe your boss think all negative would be more suitable to business for Credit Score Card ?

Re: logistic regression: sign of estimates/coefficients

ballardw — Sun, 12 Jun 2022 09:19:32 GMT

@Ksharp wrote:

I totally agree with @Rick_SAS 's opinion. I think your model fit very well .

"owever, the coefficient/estimate of one of the independent is coming out to be positive "

is due to your data , NOT from Logistic Model .

P.S. maybe your boss think all negative would be more suitable to business for Credit Score Card ?

Or maybe the sign of a variable as collected is inappropriate for use this way?

"Loss" for example might be provided as positive (as in "I lost 10 dollars", the explicit value mentioned is 10) but for a model the value should be negative (-10 effect on balance)

Re: logistic regression: sign of estimates/coefficients

adjn258 — Sun, 12 Jun 2022 12:39:22 GMT

Thanks you for all your inputs! i think you have it spot on! I was worried that maybe it is not possible statistically, that signs can be opposite. But it seems more of a worry for business usage from credit scoring view.

I have a further question - if one sign is opposite, how do I determine the weight of each factor in the final model? Currently, I am standardizing all factors (X-mean/std) and then rerunning the logistic equation - then I take the absolute value of the coefficients to arrive at the weights. I have a feeling this is probably not the best way, i am going wrong somewhere.

Re: logistic regression: sign of estimates/coefficients

PaigeMiller — Sun, 12 Jun 2022 12:55:53 GMT

I have a further question - if one sign is opposite, how do I determine the weight of each factor in the final model?

Using my understanding of "weight of each factor in the model", I would say no such thing is possible.

Currently, I am standardizing all factors (X-mean/std) and then rerunning the logistic equation - then I take the absolute value of the coefficients to arrive at the weights.

This allows you to COMPARE the regression coefficients, the largest in absolute value will have the biggest impact when a 1SD change is made.

Please understand that the coefficients are not independent of each other, and the variables in your data set are correlated with each other. So there is no real concept of a variable in the data moving 1SD while the other variables are held constant — this is theoretically possible, and mathematically possible, but it does not happen in real data sets.

Re: logistic regression: sign of estimates/coefficients

adjn258 — Sun, 12 Jun 2022 12:59:24 GMT

Thank you. But the idea of weightage (of what I am trying to derive) is - what is the relative importance of each factor in the equation, since it will be used for scoring - can you please tell me what would be the best way to arrive at that?

Re: logistic regression: sign of estimates/coefficients

adjn258 — Sun, 12 Jun 2022 13:00:21 GMT

Just to clarify more, that is, all other variables, being constant.

Re: logistic regression: sign of estimates/coefficients

adjn258 — Sun, 12 Jun 2022 13:09:20 GMT

Apologies for this question - i re-read your answer, i think you ve already answered this.

Re: logistic regression: sign of estimates/coefficients

sbxkoenk — Sun, 12 Jun 2022 13:12:54 GMT

Hello,

To determine the relative importance of each factor in the equation, you can indeed look at the (absolute value of the) standardized estimates for the parameters in the "Analysis of Maximum Likelihood Estimates" table.

But to get these standardized betas , you do not need to fit your model on standardized variables.

You can specify the STB option on the MODEL statement of PROC LOGISTIC to get these.

Also, a sign opposite to the one you expect can be caused by multicollinearity (between independent variables).
The question is : do you want your model to only predict well (in that case multicollinearity is less of a problem) or do you want your model to be a 'glass box' (focus on interpretability and explain-ability)? In the latter case, you should try to get rid of multicollinearity or reduce it to a reasonable amount.

Koen

Re: logistic regression: sign of estimates/coefficients

Ksharp — Mon, 13 Jun 2022 12:39:54 GMT

I think OP's boss want Score Card conform to the business logistic (a.k.a have better and reasonable explanation in business).
For example :
X variable (INCOME) have positive estimator, Y variable is the probability of default .
In our mind, the more INCOME should be less the probability of default (a.k.a they have negative correlation).
wheras , INCOME have positive estimator (a.k.a they have positive correlation).
wherefore, it is very hard to explain this Score Card to customer .
Why Score Card would give us a different result with real business logistic ?

Re: logistic regression: sign of estimates/coefficients

Ksharp — Mon, 13 Jun 2022 12:43:27 GMT

" if one sign is opposite, how do I determine the weight of each factor in the final model?"
I don't understand . What do you mean by "weight" ?
Once you got estimator and WOE , you could get Score Card ,no need to "standardizing all factors (X-mean/std) ".
This variable's Score also has different sign with other variables's Socre.

Re: logistic regression: sign of estimates/coefficients

Ksharp — Mon, 13 Jun 2022 12:47:17 GMT

I think CORRB option on the MODEL statement of PROC LOGISTIC could find out the multicollinearity .
As I said ,even there are no multicollinearity in MODEL ,you still could get one positive estimator ,others are negative.
It is coming from data ,not from MODEL .
Maybe OP's boss want Score Card to have more explanation in real business .that is reason OP was told to drop that positive variable.

Re: logistic regression: sign of estimates/coefficients

sbxkoenk — Mon, 13 Jun 2022 12:50:36 GMT

@Ksharp wrote:
Why Score Card would give us a different result with real business logistic ?

I guess you mean "business logic" (?).
Opposite (contra-intuitive) signs are always possible.

First check the training data and how they were collected. There you might find a reason ( aha - erlebnis ).

If training data are OK, ... I keep emphasizing, multi-collinearity is an ugly beast !

Koen

Re: logistic regression: sign of estimates/coefficients

Ksharp — Mon, 13 Jun 2022 13:07:20 GMT

I guess you mean "business logic" (?).
Opposite (contra-intuitive) signs are always possible.

That is what I mean. But for OP ,it is very hard to explain to customer ,maybe that is reason to drop that positive variable.