Programming the statistical procedures from SAS

If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

Accepted Solution Solved
Reply
Contributor
Posts: 45
Accepted Solution

If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

Thanks,

Marcio


Accepted Solutions
Solution
‎07-19-2014 10:56 PM
Respected Advisor
Posts: 4,742

Re: If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

It might be very helpful to look at the distribution of residuals from your models. It might be that you are comparing two ill-fitting models or that some outliers are wrecking havoc in the fit stats.

PG

PG

View solution in original post


All Replies
Contributor
Posts: 32

Re: If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

Normally I should think you would test for interactions first. If there are none i.e. not significant, you can assume that those terms are not important that is they do not affect the variance in your data and run models that don't have an interaction term. Does that make sense? That is my understanding. 

Contributor
Posts: 45

Re: If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

Thanks Kodmfl, but how does the AIC/BIC plays on it?

Super User
Posts: 9,758

Re: If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

It is the measure of model fitting. the lower value stand for your model is more good . as the matter of fact , I don't thing you should keep the interaction effect in the model .

Xia Keshan

Trusted Advisor
Posts: 1,215

Re: If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

These are goodness of fit measure which are used to compare one model to another.

Contributor
Posts: 45

Re: If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

So Just to make sure we are on the same page - I had a model where the interaction was not significant, but the AIC/BIC were lower in that model compared to when I took the interaction out. Based on what you all said I should take the interaction out - even though the fit is better?

Trusted Advisor
Posts: 1,215

Re: If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

I don't think there is a direct relation between AIC and significance of interaction term. If you add more predictors in the model, even some of them are irrelevant to the study you will get lower AIC.

Valued Guide
Valued Guide
Posts: 684

Re: If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

I think the comment by stat@sas is a bit misleading. If you add variables to a model, the -2 log-likelihood (-2LL) will go down. But the AIC may or may not go down. The AIC adds a penalty for the number of parameters in a model. If the addition of a variable (i.e., parameter) does not reduce the -2LL enough, then the AIC will go up. That is why the AIC is a useful measure of goodness of fit penalized by the number of parameters (it prevents drastic overfitting of a model). It is not exactly a hypothesis test, however. In programs like MIXED, GLIMMIX, etc., a scaled Wald statistic (t or F) is used to test hypotheses. This does not use the -2LL directly. The Wald test can lead to similar conclusions to that obtained by looking at AIC, but not always. There are tests possible based directly on the -2LL (likelihood ratio tests). But even these do not not exactly correspond to the AIC. You have a case where there is a disagreement. It is well known that model selection based on the AIC may lead to an overly complex model. You will have to decide on whether the interaction matters.

Trusted Advisor
Posts: 1,215

Re: If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

Thanks for the detailed explaination on AIC. As you mentioned adding a new variable in the model will reduce -2 log-likelihood. So if reduction in -2 log-likelihood by adding a new variable is higer than 2P then AIC will be on lower side right?

Valued Guide
Valued Guide
Posts: 684

Re: If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

With the addition of one parameter (e.g., one variable in a linear model), -2LL has to decrease by at least 2 for there to be a decrease in AIC.

Solution
‎07-19-2014 10:56 PM
Respected Advisor
Posts: 4,742

Re: If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

It might be very helpful to look at the distribution of residuals from your models. It might be that you are comparing two ill-fitting models or that some outliers are wrecking havoc in the fit stats.

PG

PG
Contributor
Posts: 32

Re: If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

Mario

Can you tell me a bit more about the design of your experiment? What are the data? What questions are you asking i.e. what hypotheses are you trying to test with your statistical models? It would seem to me that no matter what, you should attempt to develop the most parsimonious model possible and in that case you will probably want to drop the interaction term and rerun the analysis without the interaction if it is not significant regardless of the AIC/BIC values. Another thing to consider is not just the statistical significance but the reality of your results. If you are going to keep a particular effect in your model then you should be able to demonstrate that it exists somehow, One COULD argue, I suppose for retaining a particular term in a model (an interaction for example) at P>0.05 (but close to 0.05) if you reasonable evidence for doing so. But I don't think that better fit of your model is a good enough reason. In fact typically in my experience a statistician/scientist feels damn lucky to get rid of interactions if he can because they can be damn hard to try and explain from a theoretical point of view. That may not always be the case and interesting things CAN be discovered BECAUSE of interactions but not if they are not really there so to speak.

As for AIC and BIC they are criterion based model selection approaches that are typically used for time series and multiple regression analysis, I have not used them extensively. My advice is to understand your data as intimately as you can before allowing a statistical criterion to make choices for you.

If you are doing ANOVAs do means plots. If you are doing regressions do a 2-D or 3-D contour plot.

Hope this helps, Best of luck.

Contributor
Posts: 45

Re: If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

I appreciate all the input you gave me! That will help me deciding on the model.

Kodml - in that example the variable definitely has biological meaning.

Thanks!

Marcio

Respected Advisor
Posts: 2,655

Re: If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

Excluding based on a p value is always dangerous.  What would you be asking if one of the main effects was not significant, but the interaction was significant?  Which makes the results more interpretable?

And, what happens to corrected AIC?  That has been shown to be a better measure of model fit for smaller datasets.

Steve Denham

Contributor
Posts: 45

Re: If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

Steve, thanks I will check that!

Marcio

🔒 This topic is solved and locked.

Need further help from the community? Please ask a new question.

Discussion stats
  • 14 replies
  • 2009 views
  • 7 likes
  • 7 in conversation