12-05-2016 06:58 AM
Another question regarding logistic regression.
If you have enough volume of 'Events' does that matter how small your 'event rate' is ? I have data where Even rate is 1.3% but I have around 16K number of events in around 1.2m observations. I understand my even rate is very low but I think its enough volume to test around 20 variables in the logistic reg?
I have read few articles but honestly I am still confused if there is need to do any sampling. Many articles say if you have less than 10% even rate, you should consider oversampling but I think that is required only if you have less volume of events? Any thoughts?
12-05-2016 09:28 AM
1) try Exact Logistic Regression + Monte Carlo 2) Decision Tree / Random Forest 3)try Poisson Regression : when p~0 log(p/(1-p)) ~ log(p) = log(r/n) --> log(r)-log(n) and move log(n) into right of model. http://support.sas.com/kb/24/188.html
12-05-2016 09:41 AM
Sorry, I tried to read this but it did not ans my question. If its about SAS showing significant results or confidence intervals, then yes my results are showing that.
What I am trying to understand is:
If I have enough volume of Events does the event rate matter? I have 16K Event out of 1.2 million obs and event rate of approx 1.3%.
12-05-2016 10:02 AM
No. It is not good for Logistic Model because event rate is too low . So it doesn't matter how many obs you have.
12-05-2016 04:33 PM
12-05-2016 11:04 PM
Hi, actually I am not expert about statistic . proc logistic model the probability of event ,not the number of event. It is called overdispersion . In the sas documentation of logistic has described it. Overdispersion For a correctly specified model, the Pearson chi-square statistic and the deviance, divided by their degrees of freedom, should be approximately equal to one. When their values are much larger than one, the assumption of binomial variability might not be valid and the data are said to exhibit overdispersion. Underdispersion, which results in the ratios being less than one, occurs less often in practice. When fitting a model, there are several problems that can cause the goodness-of-fit statistics to exceed their degrees of freedom. Among these are such problems as outliers in the data, using the wrong link function, omitting important terms from the model, and needing to transform some predictors. These problems should be eliminated before proceeding to use the following methods to correct for overdispersion.