topic Re: Akaike's Information Criterion vs proc mixed - how to convince that the best model has merit? in Statistical Procedures

Akaike's Information Criterion vs proc mixed - how to convince that the best model has merit?

LudekBartos — Wed, 04 Aug 2021 17:40:33 GMT

Dear all,

I started using Akaike's Information Criterion. One of the key papers concerned with this approach says, "AIC ranks the models in the set of alternatives; if none have merit, the models are still ranked. Thus, one needs some measure of the ‚worth' of either the global model or the model estimated to be best."... "Thus, standard statistical methods are needed to gauge this matter; these include adjusted R2, goodness-of-fit tests, and the analysis of regression residuals." (Burnham KP, Anderson DR, Huyvaert KP 2011. AIC model selection and multimodel inference in behavioral ecology: some background, observations, and comparisons. Behavioral Ecology and Sociobiology 65: 23-35)
I have used the PROC MIXED (with random and fixed effects). However, so far as I know, PROC MIXED supports neither adjusted R2, nor goodness-of-fit tests. Any advice on what to use instead? Or how to convince readers of the paper that the best candidate model has merit?

Re: Akaike's Information Criterion vs proc mixed - how to convince that the best model has merit?

sbxkoenk — Wed, 04 Aug 2021 21:55:01 GMT

Hello,

See here (same question as yours):

Lack of fit test in Proc Mixed?

https://communities.sas.com/t5/Statistical-Procedures/Lack-of-fit-test-in-Proc-Mixed/td-p/177767

Regards,

Koen

Re: Akaike's Information Criterion vs proc mixed - how to convince that the best model has merit?

pmbrown — Thu, 05 Aug 2021 08:39:43 GMT

discussion here re IC and BIC may be relevant: https://communities.sas.com/t5/Statistical-Procedures/Some-issues-with-BIC-calculations-in-Proc-Mixed/m-p/759105#M36967

Re: Akaike's Information Criterion vs proc mixed - how to convince that the best model has merit?

SteveDenham — Thu, 05 Aug 2021 15:13:37 GMT

There are several pseudo-Rsquared values out there, so Google is your friend for that.

To see if any of the models are doing anything, you could calculate the AIC (or corrected AIC) for the null model (no fixed effects as the random effects are likely design effects), and look at the relative likelihood [=exp((AIC1 - AIC2)/2). Also, since the model is nested within the null model, a likelihood ratio test would be appropriate. You should use ML rather than REML to compare models that differ in the fixed effects.

SteveDenham

Re: Akaike's Information Criterion vs proc mixed - how to convince that the best model has merit?

LudekBartos — Fri, 06 Aug 2021 19:42:45 GMT

If I understand it well, do you suggest simply using the Akaike Weight, wi for the null model?

Thanks a lot,

Ludek