topic Re: Mean Square Error in SAS Data Science

Mean Square Error

NicolasC — Fri, 15 Sep 2017 08:04:02 GMT

Hi there

I know this topic has already been adressed but it did not fully answer my question. And I do not seem to find the answer on the Miner Help window.

I am using Model Comparison node to compare different models. Target is interval, and inputs are a mix of nominal-interval-binary. I see two options to compare my models: Averge Squared Error (AVE) and Mean Squared Error (MSE). I am confused with the terminology. My understanding is that MSE=SSE/DFE, with SSE the Error Sum of Squares and DFE the Error degrees of freedom, with DFE=n-p-1, with n number of obervations and p number of variables used in the model. Is AVE Miner definition of SSE? If so, the 'Average' is confusing as SSE is just a sum of squared differences. I also do not understand why (see pic attached) MSE is calculated for one model only when I choose this option. Also, the models should anyway be ranked the same order as long as DFE is the same right?

Jumping on another topic, MSE comparion is what I aim for lately as I will want to compare models, some runing on the same database as before, some running on the same database but with reduced number of variables p (and obviously same number of observations n). Since I want to account for this change in my assesment of models, MSE does that.

And finally, is there a way in Miner to calculate SST (Total Sum of Squares) and SSM (Model Sum of Squares) so that I can myself calculate the coefficient of determination (asuming there is no option to do so directly for R2??)

Many thanks

Nicolas

Re: Mean Square Error

NicolasC — Fri, 15 Sep 2017 08:21:15 GMT

Following my previous post, it seems that the AVE=Sum of Squared Errors/Divisor for ASE. So AVE is not SSE but SSE divided by a divisor of some kind. Where does this divisor number comes from? I have n=200,000+ and p circa 250 so I am intrigued how to combine those (if this is the way to do it) to get a divisor in the range of 7,000. Many thanks

Re: Mean Square Error

BrettWujek — Fri, 15 Sep 2017 12:18:17 GMT

Hey Nicolas - Please see the following post on this. I think it explains the divisor fairly well.

https://communities.sas.com/t5/SAS-Data-Mining-and-Machine/Mean-Squared-Error-vs-Average-Squared-Error/m-p/284742/highlight/true#M4223

Re: Mean Square Error

NicolasC — Sat, 16 Sep 2017 15:53:32 GMT

Thanks for your answer. I had gone through this post before opening a new topic. This post does not address the different questions I mentionned.