topic Re: Use of ASE to assess model complexity in SAS Academy for Data Science

Use of ASE to assess model complexity

pvareschi — Sat, 02 May 2020 18:05:11 GMT

Perhaps this is something well known to everybody but to me it has been a surprise: after doing some tests on fitting models with and without defining prior probabilities I have noticed that Enterprise Miner does not take account of prior probabilities when calculating Average Square Error (ASE) (same applies to the calculation of residuals as saved on output datasets)

That being the case, I just want to clarify whether there is any chance/scenario under which we would end up choosing a different model (out of a sequence of models of increasing complexity - e.g. Regression) if ASE was indeed adjusted for prior probabilities.

My instinct tells me that is not the case, but I wonder whether there is a more mathematical justification for that.

Re: Use of ASE to assess model complexity

pvareschi — Mon, 04 May 2020 09:16:13 GMT

Just to further clarify, I am referring to "Applied Analytics Using SAS Enterprise Miner", "Lesson 7: Model Assessment Using SAS Enterprise Miner", "Adjusting for Separate Sampling": if we do not specify prior probabilities, we know that performance metrics are inaccurate and/or biased; however, what I am concerned is whether it would affect the choice of the "best model", especially when applied to a single modelling node to assess model complexity. My understanding is that it would not be the case, at least when using ASE or misclassification rate (Profit/Loss would be a different matter)

Re: Use of ASE to assess model complexity

gcjfernandez — Wed, 06 May 2020 18:44:00 GMT

I agree with your comments because adjusting for prior probability basically only shifting the intercept values. Therefore this should not affect the model selection. However, if you want the prior values affect your model decision you should consider the decision option and provide decision weights (Please refer Chapter 6 in the AAEM course notes)

Re: Use of ASE to assess model complexity

pvareschi — Fri, 15 May 2020 05:10:42 GMT

Just a further clarification on the statement "because adjusting for prior probability basically only shifting the intercept values": that is true for linear models (i.e. Logistic Regression); but what about non-parametric or non-linear models such as Decision Trees and Neural Networks? Would that still just result in a shift of the intercept values?

Re: Use of ASE to assess model complexity

gcjfernandez — Sat, 16 May 2020 17:37:59 GMT

Your question:

When the target variable is binary we call this predictive model a classification model and the goal of Decision tree, Logistic regression, or NN is to classy the binary target correctly. All these models create all possible pairs of one event and one non event and if these models correctly classify one pair at a time then they are called concordance pair. Otherwise discordance pair. Therefore by random chance there is 50% chance of finding the event within a pair. We hope that the model we develop will have a higher chance of differentiating event from the non event. These statistics (% of concordance and discordance) are the basis of ROC index. ROC index is not influenced by the Prior probability. Therefore ROC index is a popular model comparison statistics. Also the proportion of events to non events in the population is not considered when developing classification models by default.

However, if the goal of scoring is computing posterior probabilities, then the posterior probabilities needs to be adjusted for prior probability after we develop the model. This adjustment will be the same whether we use Decision tree (base line adjustment ), logistic regression or NN(Intercept, offset, bias). Because in this prior probability adjustment (non-linear component of the model is not included).

I hope this explanation is adequate

Re: Use of ASE to assess model complexity

pvareschi — Thu, 21 May 2020 15:38:43 GMT

Thank you for your explanation; very thorough!