Building models with SAS Enterprise Miner, SAS Factory Miner, SAS Visual Data Mining and Machine Learning or just with programming

Actual vs Predicted , help please. Thank You

Reply
Frequent Contributor
Posts: 95

Actual vs Predicted , help please. Thank You


Hi All,

I have built a logistic regression model on the training dataset, then score my validation dataset. I have calculated the Mean Absolute Percentage Error  between Actual and Predicted (Table below) Is there any rule to say if my MAPE is > 10% for example, the model doesn't generalize well, so need to improve the model or build a new one? I would like to set up a rule. Your help would be much appreciated,

Many Thanks

Decile GroupActualPredictedAbsolute MAPE
(Mean Absolute Percentage Error)
105886012%
94944971%
84724426%
74024030%
63923745%
53273476%
43033205%
33162976%
22672712%
12322424%
Respected Advisor
Posts: 2,655

Re: Actual vs Predicted , help please. Thank You

Any rule is going to depend on the relative cost of making an error.  In some fields, an error >10% may be a good cutpoint, but in others, it may need to be substantially less.  Also, percentages are notorious for being scale dependent.  In your example, a deviation of 13 with a denominator of 588, leads to a percentage error of 2%.  However, suppose your model predicted a deviation of only 1.3 (much better), but the denominator was only 5.88.  Now the percentage error is 22%, even though the absolute prediction is a magnitude more precise.

That probably doesn't answer your question of HOW to set a cutpoint in EM, but it is something to consider when the question is put forward.

Steve Denham

Ask a Question
Discussion stats
  • 1 reply
  • 474 views
  • 0 likes
  • 2 in conversation