Actual vs Predicted , help please. Thank You

[Kanyange]

Hi All,

I have built a logistic regression model on the training dataset, then score my validation dataset. I have calculated the Mean Absolute Percentage Error  between Actual and Predicted (Table below) Is there any rule to say if my MAPE is > 10% for example, the model doesn't generalize well, so need to improve the model or build a new one? I would like to set up a rule. Your help would be much appreciated,

Many Thanks

Decile Group	Actual	Predicted	Absolute MAPE (Mean Absolute Percentage Error)
10	588	601	2%
9	494	497	1%
8	472	442	6%
7	402	403	0%
6	392	374	5%
5	327	347	6%
4	303	320	5%
3	316	297	6%
2	267	271	2%
1	232	242	4%

[SteveDenham]

Any rule is going to depend on the relative cost of making an error.  In some fields, an error >10% may be a good cutpoint, but in others, it may need to be substantially less.  Also, percentages are notorious for being scale dependent.  In your example, a deviation of 13 with a denominator of 588, leads to a percentage error of 2%.  However, suppose your model predicted a deviation of only 1.3 (much better), but the denominator was only 5.88.  Now the percentage error is 22%, even though the absolute prediction is a magnitude more precise.

That probably doesn't answer your question of HOW to set a cutpoint in EM, but it is something to consider when the question is put forward.

Steve Denham