Hi All,
I have built a logistic regression model on the training dataset, then score my validation dataset. I have calculated the Mean Absolute Percentage Error between Actual and Predicted (Table below) Is there any rule to say if my MAPE is > 10% for example, the model doesn't generalize well, so need to improve the model or build a new one? I would like to set up a rule. Your help would be much appreciated,
Many Thanks
Decile Group | Actual | Predicted | Absolute MAPE (Mean Absolute Percentage Error) |
10 | 588 | 601 | 2% |
9 | 494 | 497 | 1% |
8 | 472 | 442 | 6% |
7 | 402 | 403 | 0% |
6 | 392 | 374 | 5% |
5 | 327 | 347 | 6% |
4 | 303 | 320 | 5% |
3 | 316 | 297 | 6% |
2 | 267 | 271 | 2% |
1 | 232 | 242 | 4% |
Any rule is going to depend on the relative cost of making an error. In some fields, an error >10% may be a good cutpoint, but in others, it may need to be substantially less. Also, percentages are notorious for being scale dependent. In your example, a deviation of 13 with a denominator of 588, leads to a percentage error of 2%. However, suppose your model predicted a deviation of only 1.3 (much better), but the denominator was only 5.88. Now the percentage error is 22%, even though the absolute prediction is a magnitude more precise.
That probably doesn't answer your question of HOW to set a cutpoint in EM, but it is something to consider when the question is put forward.
Steve Denham
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!
Use this tutorial as a handy guide to weigh the pros and cons of these commonly used machine learning algorithms.
Find more tutorials on the SAS Users YouTube channel.