Building models with SAS Enterprise Miner, SAS Factory Miner, SAS Visual Data Mining and Machine Learning or just with programming

Help, Attrition Model Performance in SAS. Thanks

Reply
Frequent Contributor
Posts: 96

Help, Attrition Model Performance in SAS. Thanks

Hi,

I have build an attrition model and I am evaluating its perfomance. I have sorted the probabilities from high to low, dividing the customers into ten
equally-sized groups called “deciles”, such that ten percent of the customer base is con-tained in each decile, and observing model performance in terms of attrition rate by decile. Using the code below..

proc rank data=OUT groups=10 out=OUT_DECILE descending;

var P; ranks decile;

run;

data OUT_DECILE;

set OUT_DECILE;

decile=decile+1;

run;

proc means data=OUT_DECILE n mean sum;

var LAPSE;

class decile;

run;

I have run it first on the training dataset used to build model and I get this below. Then I have scored the validation dataset, and rank the probabilities again and I get this below. Shall I not get roughly the same % per decile? Is my model not performing well then? I have used the gain chart to compare Validation and Trainig but they looked fine? Your help woul be much appreciated . Many Thanks

Analysis Variable  - LAPSE : Training Sample
Rank for VariableN ObsDecile MeanOverall Mean
pred
120,98679%30%
221,01470%30%
320,99938%30%
421,04129%30%
517,83925%30%
624,16822%30%
720,95220%30%
821,01312%30%
920,9987%30%
1020,9905%30%

Analysis Variable : LAPSE : Validation Sample
Rank for VariableN ObsDecile MeanOverall Mean
pred
19,034100%21%
29,034100%21%
39,03413%21%
48,9680%21%
510,8220%21%
67,3120%21%
79,0330%21%
89,0140%21%
99,0550%21%
109,0340%21%

I hve attached the gain chart. Many Thanks

Occasional Contributor
Posts: 17

Re: Help, Attrition Model Performance in SAS. Thanks

Knowing nothing else, it seems to me that your training model is not generalizing well to the validation set. Which is usually a sign of overfitting.

What tool are you using to create the initial model, and what technique?

Frequent Contributor
Posts: 96

Re: Help, Attrition Model Performance in SAS. Thanks

Posted in reply to adjgiulio

Hi,

I am using SAS and Logistic Regression. But the gain chart is showing that the model is robust. Please See attached

Many Thanks

Alice

Frequent Contributor
Posts: 99

Re: Help, Attrition Model Performance in SAS. Thanks

Hi,                 Did you use the train decile definition for validation data. For example for training data, the first decile the min and max probabilities were say 0.9 - 0.95. Then you should use the same decile definition for Validation data.                  If you have used great, else use the train dataa decile definitions to compare train with validation.        Best Regars,      Amit
Ask a Question
Discussion stats
  • 3 replies
  • 556 views
  • 0 likes
  • 3 in conversation