Building models with SAS Enterprise Miner, SAS Factory Miner, SAS Visual Data Mining and Machine Learning or just with programming

GLM not predicting lower values

Reply
Occasional Contributor
Posts: 5

GLM not predicting lower values

Hi,
I am trying to predict an interval target (customer spend in $) using GLM in SAS EM 13.2. I have 80K observations and 300 variables.
The problem that I have with the final model is that the minimum prediction for the target is around $50 whereas 11K of customers have spent less than $50 in the training dataset and 1% of customers have spent less than $1.

Any thoughts on why is this happening or on how to fix it?
The probability distribution that I have used in the GLM model is Gamma with a Log link function. I have also tried other prob. distributions as well as link functions like Tweedie or invert gaussian but Gamma with a log link function produced the smallest ASE. The distribution of Target variable is highly skewed on the right as there are plenty of customers spending lower amounts and only a few spending more than $1000.

Any thoughts are highly appreciated,
Thanks,
Sam

Contributor
Posts: 64

Re: GLM not predicting lower values

Sometimes you are dealing with two different sets of underlying drivers, so something that might work is to see if you can identify those who spend less than $50 using a binary regression model first. If you can predict those (meaning you are getting a strong model), simply run the binary model first, and then run a separate GLM for each. I'll bet you can see which variables are different by comparing the GLM results.
Occasional Contributor
Posts: 5

Re: GLM not predicting lower values

Thank you JBerry,

 

I really liked your idea but I am still confused that why GLM is unable to predict those small spends even without segmenting the whole population.

I have already segmented the population based on the credit limit of customers: CL le $500 and CL g $500.

 

Thanks,

Sam

Ask a Question
Discussion stats
  • 2 replies
  • 314 views
  • 1 like
  • 2 in conversation