01-08-2014 11:26 AM
I have built a logistic regression model on the training dataset, then score my validation dataset. I have calculated the Mean Absolute Percentage Error between Actual and Predicted (Table below) Is there any rule to say if my MAPE is > 10% for example, the model doesn't generalize well, so need to improve the model or build a new one? I would like to set up a rule. Your help would be much appreciated,
|Decile Group||Actual||Predicted||Absolute MAPE|
(Mean Absolute Percentage Error)
01-08-2014 12:45 PM
Any rule is going to depend on the relative cost of making an error. In some fields, an error >10% may be a good cutpoint, but in others, it may need to be substantially less. Also, percentages are notorious for being scale dependent. In your example, a deviation of 13 with a denominator of 588, leads to a percentage error of 2%. However, suppose your model predicted a deviation of only 1.3 (much better), but the denominator was only 5.88. Now the percentage error is 22%, even though the absolute prediction is a magnitude more precise.
That probably doesn't answer your question of HOW to set a cutpoint in EM, but it is something to consider when the question is put forward.