topic Re: Model Validation. Please Help.. Thank you in SAS Data Science

Model Validation. Please Help.. Thank you

Kanyange — Thu, 05 Dec 2013 16:39:29 GMT

Hi,

I have built a logistic regression model to predict customers who are more likely to click on a marketing offer. My model is not great as I have a AUC=0.58 and 50% capture only 60%. So I have built the model on a training sample (9 months) then kept 2 months of data as sample held out from model dev (Validation) to test my prediction. Once I finished to build the model on the training, I have scored the validation (2 months dataset) then sorted the scored validation, and sliced the target variable and probabilities into deciles ..like below...It doesn't seem that my model is ranking well ...Is it because the AUC is low..so the predictive power is not great? Your help will be much appreciated....Many Thanks

proc rank data=VALIDATION (KEEP=CLICK_FLAG EM_EVENTPROBABILITY) groups=10 out=pred_valid;

var EM_EVENTPROBABILITY; ranks decile;

run;

data pred_valid;

set pred_valid;

decile=decile+1;

run;

proc sort data=pred_valid;

by descending EM_EVENTPROBABILITY;

run;

proc means data=pred_valid n mean sum;

var CLICK_FLAG;

class decile;

run;

proc means data=pred_validn mean sum;

var EM_EVENTPROBABILITY;

class decile;

run;

Decile	Training Response Rate	Training Mean_Score	Validation Response Rate	Validation Mean_Score
10	10%	0.10	6%	0.10
9	9%	0.08	6%	0.08
8	7%	0.07	7%	0.07
7	6%	0.06	7%	0.06
6	6%	0.06	7%	0.06
5	6%	0.06	7%	0.06
4	6%	0.05	7%	0.05
3	5%	0.05	6%	0.05
2	5%	0.05	5%	0.05
1	4%	0.03	7%	0.03

Re: Model Validation. Please Help.. Thank you

jwexler — Fri, 06 Dec 2013 21:16:39 GMT

Kanyange, would you mind sharing your modeling code? It would help to understand the options you selected, along with a snapshot of the output.

Thanks,

Jonathan

Re: Model Validation. Please Help.. Thank you

Kanyange — Mon, 09 Dec 2013 13:55:31 GMT

Hi Jonathan,

Thank you for your reply,...yes sure , I have attached the Code and the output...Please let me know if you can open the attachement,....

Cheers

Re: Model Validation. Please Help.. Thank you

jwexler — Thu, 12 Dec 2013 14:23:38 GMT

Kanyange, this question would be best posted in the SAS Stat Community

Given your needs with PROC LOGISTIC, they would be best to respond appropriately.

Let me know if you need any more help.

Thanks,

Jonathan