Akaike's Information Criterion vs proc mixed - how to convince that the best model has merit?

[LudekBartos]

Dear all,

I started using Akaike's Information Criterion. One of the key papers concerned with this approach says, "AIC ranks the models in the set of alternatives; if none have merit, the models are still ranked. Thus, one needs some measure of the ‚worth' of either the global model or the model estimated to be best."... "Thus, standard statistical methods are needed to gauge this matter; these include adjusted R2, goodness-of-fit tests, and the analysis of regression residuals." (Burnham KP, Anderson DR, Huyvaert KP 2011. AIC model selection and multimodel inference in behavioral ecology: some background, observations, and comparisons. Behavioral Ecology and Sociobiology 65: 23-35)
I have used the PROC MIXED (with random and fixed effects). However, so far as I know, PROC MIXED supports neither adjusted R2, nor goodness-of-fit tests. Any advice on what to use instead? Or how to convince readers of the paper that the best candidate model has merit?

[sbxkoenk]

Hello,

See here (same question as yours):

Lack of fit test in Proc Mixed?

https://communities.sas.com/t5/Statistical-Procedures/Lack-of-fit-test-in-Proc-Mixed/td-p/177767

Regards,

Koen

[pmbrown]

discussion here re IC and BIC may be relevant: https://communities.sas.com/t5/Statistical-Procedures/Some-issues-with-BIC-calculations-in-Proc-Mixe...

[SteveDenham]

There are several pseudo-Rsquared values out there, so Google is your friend for that.

To see if any of the models are doing anything, you could calculate the AIC (or corrected AIC) for the null model (no fixed effects as the random effects are likely design effects), and look at the relative likelihood [=exp((AIC1 - AIC2)/2).  Also, since the model is nested within the null model, a likelihood ratio test would be appropriate.  You should use ML rather than REML to compare models that differ in the fixed effects.

SteveDenham

[LudekBartos · Accepted solution]

If I understand it well, do you suggest simply using the Akaike Weight, wi for the null model?

Thanks a lot,

Ludek