topic Re: How to transform right skewed data with heavy concentration at zero value? in SAS Data Science

How to transform right skewed data with heavy concentration at zero value?

anu1999 — Tue, 15 Mar 2016 18:14:10 GMT

Hi,

I am working with a data with 50+ variables as input to the model. Most of the input variables have the right skewed distribution heavy at 0 value. I know log transformation is used for right skewed data but in my case when I have high concentration of zero, it wont help.

I was wondering if you have any suggestions based on your past experience on how you dealt with such data for modeling.

Thanks
A

Re: How to transform right skewed data with heavy concentration at zero value?

Ksharp — Wed, 16 Mar 2016 01:40:19 GMT

Did your data confirm Possion distribution ?

You could check Zero Inflation Model.

Re: How to transform right skewed data with heavy concentration at zero value?

anu1999 — Thu, 17 Mar 2016 17:46:55 GMT

Hi Xia,

Thanks for suggestion. After reasearching, it looks like Zero Inflation model is the right choice here. Do you know if this can be implemented in SAS enterprise miner.

Thanks.

Re: How to transform right skewed data with heavy concentration at zero value?

WendyCzika — Thu, 17 Mar 2016 19:36:13 GMT

You can fit a GLM with the zero-inflated Poisson distribution in the HP GLM node in Enterprise Miner (in releases 13.1 and beyond). But that's for a target that has many 0's. For inputs that are skewed, you could still use a Log transformation, just need to add a constant to the variables first to be able to log 0. The Transform Variables node in EM can do the log transformation and will automatically add a constant.