topic Actual vs Predicted , help please. Thank You in SAS Data Science

Actual vs Predicted , help please. Thank You

Kanyange — Wed, 08 Jan 2014 16:26:11 GMT

Hi All,

I have built a logistic regression model on the training dataset, then score my validation dataset. I have calculated the Mean Absolute Percentage Error between Actual and Predicted (Table below) Is there any rule to say if my MAPE is > 10% for example, the model doesn't generalize well, so need to improve the model or build a new one? I would like to set up a rule. Your help would be much appreciated,

Many Thanks

Decile Group	Actual	Predicted	Absolute MAPE (Mean Absolute Percentage Error)
10	588	601	2%
9	494	497	1%
8	472	442	6%
7	402	403	0%
6	392	374	5%
5	327	347	6%
4	303	320	5%
3	316	297	6%
2	267	271	2%
1	232	242	4%

Re: Actual vs Predicted , help please. Thank You

SteveDenham — Wed, 08 Jan 2014 17:45:20 GMT

Any rule is going to depend on the relative cost of making an error. In some fields, an error >10% may be a good cutpoint, but in others, it may need to be substantially less. Also, percentages are notorious for being scale dependent. In your example, a deviation of 13 with a denominator of 588, leads to a percentage error of 2%. However, suppose your model predicted a deviation of only 1.3 (much better), but the denominator was only 5.88. Now the percentage error is 22%, even though the absolute prediction is a magnitude more precise.

That probably doesn't answer your question of HOW to set a cutpoint in EM, but it is something to consider when the question is put forward.

Steve Denham