Building models with SAS Enterprise Miner, SAS Factory Miner, SAS Visual Data Mining and Machine Learning or just with programming

Model Validation. Please Help.. Thank you

Reply
Frequent Contributor
Posts: 95

Model Validation. Please Help.. Thank you

Hi,

I have built a logistic regression model to predict customers who are more likely to click on a marketing offer. My model is not great as I have a AUC=0.58 and 50% capture only 60%. So I have built the model on a training sample  (9 months) then kept 2 months of data as sample held out from model dev (Validation) to test my prediction. Once I finished to build the model on the training, I have scored the validation (2 months dataset) then sorted the scored validation, and sliced the target variable and probabilities into deciles ..like below...It doesn't seem that my model is ranking well ...Is it because the AUC is low..so the predictive power is not great?  Your help will be much appreciated....Many Thanks

proc rank data=VALIDATION (KEEP=CLICK_FLAG EM_EVENTPROBABILITY) groups=10 out=pred_valid;

  var EM_EVENTPROBABILITY; ranks decile;

run;

data pred_valid;

  set pred_valid;

decile=decile+1;

run;

proc sort data=pred_valid;

by descending EM_EVENTPROBABILITY;

run;

proc means data=pred_valid n mean sum;

  var CLICK_FLAG;

  class decile;

run;

proc means data=pred_validn mean sum;

var EM_EVENTPROBABILITY;

class decile;

run;

DecileTraining Response RateTraining Mean_ScoreValidation Response RateValidation Mean_Score
1010%0.106%0.10
99%0.086%0.08
87%0.077%0.07
76%0.067%0.06
66%0.067%0.06
56%0.067%0.06
46%0.057%0.05
35%0.056%0.05
25%0.055%0.05
14%0.037%0.03
SAS Employee
Posts: 68

Re: Model Validation. Please Help.. Thank you

Kanyange, would you mind sharing your modeling code?  It would help to understand the options you selected, along with a snapshot of the output.

Thanks,

Jonathan

Frequent Contributor
Posts: 95

Re: Model Validation. Please Help.. Thank you

Hi Jonathan,

Thank you for your reply,...yes sure , I have attached the Code and the output...Please let me know if you can open the attachement,....

Cheers

SAS Employee
Posts: 68

Re: Model Validation. Please Help.. Thank you

Kanyange, this question would be best posted in the SAS Stat Community

Given your needs with PROC LOGISTIC, they would be best to respond appropriately.

Let me know if you need any more help.

Thanks,

Jonathan

Ask a Question
Discussion stats
  • 3 replies
  • 504 views
  • 0 likes
  • 2 in conversation