topic Re: Proc Mixed - R-Squared in Statistical Procedures

Proc Mixed - R-Squared

bnd — Fri, 03 Apr 2020 21:39:36 GMT

Hello,

Is there a way to calculate the r-squared or pseudo-r-squared for proc mixed in SAS (models with fixed and random effects)? Or would it have to be hand calculated?

I saw one post that stated to run the null model and then the full-model and to look at the variance components.

Any other thoughts, suggestions, or clarifications as to how to best calculate the r-squared when using proc mixed?

Thanks!

Re: Proc Mixed - R-Squared

PaigeMiller — Sat, 04 Apr 2020 11:34:42 GMT

There is no generally agreed upon way to compute R-squared for generalized linear models, such as PROC MIXED. A number of methods have been proposed, these all have certain advantages and certain disadvantages. Your favorite search engine will find many discussions about this.

Re: Proc Mixed - R-Squared

bnd — Wed, 08 Apr 2020 00:51:16 GMT

Hi @PaigeMiller

Thanks! There are many ways to compute the R-squared for multilevel models. I think I have found one that works well.

Re: Proc Mixed - R-Squared

SteveDenham — Wed, 08 Apr 2020 11:30:31 GMT

Would you care to share that method with the rest of us?

Thanks in advance.

SteveDenham

Re: Proc Mixed - R-Squared

PaigeMiller — Wed, 08 Apr 2020 11:32:34 GMT

@bnd wrote:

Hi @PaigeMiller

Thanks! There are many ways to compute the R-squared for multilevel models. I think I have found one that works well.

Yes, I agree with @SteveDenham, you need to explain what method you chose, and why, so we can all learn.

Re: Proc Mixed - R-Squared

bnd — Fri, 10 Apr 2020 02:47:51 GMT

Hi @SteveDenham and @PaigeMiller,

I chose this formula, R-squared = 1 - SSE_Model / SSE_IntOnly. SSE represents the sum of squared residuals from the model and SSE_IntOnly represents the sum of squared residuals from the intercept-only model. I chose this model because I was looking for a simple and less complicated formula to calculate the percent reduction in variance from the null model to the full model. I used the covariance parameter estimates table from proc mixed to calculate the R-squared. http://math.usu.edu/jrstevens/stat5200/25.Rsquare_Design.pdf

I am not sure if I have explained this well! I am very new to calculating the R-squared for multilevel models. I am not sure if this approach is the best or if R-squared should even be calculated this way, but it was a simple formula for me.

I also found this formula, R-squared = SSR/CTSS, where the SSR is the reduction sums of squares due to the model over and above the mean and the CTSS is the corrected total sum of squares. I got the same percent reductions using this formula. http://animsci.agrenv.mcgill.ca/StatisticalMethodsII/drvpseudor.pdf

Re: Proc Mixed - R-Squared

SteveDenham — Fri, 10 Apr 2020 14:59:56 GMT

While this works, remind yourself over and over that the sums of squares in a mixed model are NOT what is optimized. It is a maximum likelihood method, and only in the fully balanced design with uncorrelated errors would the sums of squares be the same. A good substitute might be to look at the AIC values and determine the amount of information retained from the null model in the fit model. You could even put this on a relative basis. See the Wikipedia article on Akaike Information Criterion https://en.wikipedia.org/wiki/Akaike_information_criterion , which is a very good summary and points out how to compare models and the caveats involved.

SteveDenham

Re: Proc Mixed - R-Squared

bnd — Fri, 10 Apr 2020 23:19:49 GMT

Hi @SteveDenham,

I actually looked at the AIC as well! Maybe I should just focus on the AIC instead of the pseudo-R-squared because as you have stated the sum of squares is not what is being optimized in mixed models.

Thanks!

Brittney

Re: Proc Mixed - R-Squared

zjppdozen — Thu, 29 Apr 2021 14:57:08 GMT

Hi Steve,

Do you think likelihood ratio r-squared will be a better pseudo-r2 in mixed model, as described here https://www.ars.usda.gov/ARSUserFiles/80000000/SpatialWorkshop/19kramersupplrsq.pdf?

Also, the formula for likelihood ratio r-squared is Rlr = 1-exp(-2/n(LLM-LL0)). In the longitudinal data, do you know the "n" here should be the total person-year observations or just the total subjects included in the data set?

Re: Proc Mixed - R-Squared

SteveDenham — Fri, 30 Apr 2021 16:44:30 GMT

The Kramer paper looks quite good, and I can see some utility in the MLE based pseudo-R2. However, you would have to be sure to change to an ML method from the standard REML methods used in MIXED and GLIMMIX, and that leads to biased estimates (as a simple example, compare the biased estimate of the variance (denominator=n) to the unbiased estimate (denominator = n-1), the proof that the biased estimate is the ML estimate is a pretty standard math stats course proof). I think we are still looking for an appropriate approach to goodness of fit for REML mixed models.

BTW, the n in the formula is the total number of observations.

SteveDenham