BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
marcioadg
Calcite | Level 5

If the interaction is not significant but the model has a better fit (AIC/BIC), should I keep the interaction in the model?

Thanks,

Marcio

1 ACCEPTED SOLUTION

Accepted Solutions
PGStats
Opal | Level 21

It might be very helpful to look at the distribution of residuals from your models. It might be that you are comparing two ill-fitting models or that some outliers are wrecking havoc in the fit stats.

PG

PG

View solution in original post

14 REPLIES 14
kodmfl
Obsidian | Level 7

Normally I should think you would test for interactions first. If there are none i.e. not significant, you can assume that those terms are not important that is they do not affect the variance in your data and run models that don't have an interaction term. Does that make sense? That is my understanding. 

Ksharp
Super User

It is the measure of model fitting. the lower value stand for your model is more good . as the matter of fact , I don't thing you should keep the interaction effect in the model .

Xia Keshan

stat_sas
Ammonite | Level 13

These are goodness of fit measure which are used to compare one model to another.

marcioadg
Calcite | Level 5

So Just to make sure we are on the same page - I had a model where the interaction was not significant, but the AIC/BIC were lower in that model compared to when I took the interaction out. Based on what you all said I should take the interaction out - even though the fit is better?

stat_sas
Ammonite | Level 13

I don't think there is a direct relation between AIC and significance of interaction term. If you add more predictors in the model, even some of them are irrelevant to the study you will get lower AIC.

lvm
Rhodochrosite | Level 12 lvm
Rhodochrosite | Level 12

I think the comment by stat@sas is a bit misleading. If you add variables to a model, the -2 log-likelihood (-2LL) will go down. But the AIC may or may not go down. The AIC adds a penalty for the number of parameters in a model. If the addition of a variable (i.e., parameter) does not reduce the -2LL enough, then the AIC will go up. That is why the AIC is a useful measure of goodness of fit penalized by the number of parameters (it prevents drastic overfitting of a model). It is not exactly a hypothesis test, however. In programs like MIXED, GLIMMIX, etc., a scaled Wald statistic (t or F) is used to test hypotheses. This does not use the -2LL directly. The Wald test can lead to similar conclusions to that obtained by looking at AIC, but not always. There are tests possible based directly on the -2LL (likelihood ratio tests). But even these do not not exactly correspond to the AIC. You have a case where there is a disagreement. It is well known that model selection based on the AIC may lead to an overly complex model. You will have to decide on whether the interaction matters.

stat_sas
Ammonite | Level 13

Thanks for the detailed explaination on AIC. As you mentioned adding a new variable in the model will reduce -2 log-likelihood. So if reduction in -2 log-likelihood by adding a new variable is higer than 2P then AIC will be on lower side right?

lvm
Rhodochrosite | Level 12 lvm
Rhodochrosite | Level 12

With the addition of one parameter (e.g., one variable in a linear model), -2LL has to decrease by at least 2 for there to be a decrease in AIC.

PGStats
Opal | Level 21

It might be very helpful to look at the distribution of residuals from your models. It might be that you are comparing two ill-fitting models or that some outliers are wrecking havoc in the fit stats.

PG

PG
kodmfl
Obsidian | Level 7

Mario

Can you tell me a bit more about the design of your experiment? What are the data? What questions are you asking i.e. what hypotheses are you trying to test with your statistical models? It would seem to me that no matter what, you should attempt to develop the most parsimonious model possible and in that case you will probably want to drop the interaction term and rerun the analysis without the interaction if it is not significant regardless of the AIC/BIC values. Another thing to consider is not just the statistical significance but the reality of your results. If you are going to keep a particular effect in your model then you should be able to demonstrate that it exists somehow, One COULD argue, I suppose for retaining a particular term in a model (an interaction for example) at P>0.05 (but close to 0.05) if you reasonable evidence for doing so. But I don't think that better fit of your model is a good enough reason. In fact typically in my experience a statistician/scientist feels damn lucky to get rid of interactions if he can because they can be damn hard to try and explain from a theoretical point of view. That may not always be the case and interesting things CAN be discovered BECAUSE of interactions but not if they are not really there so to speak.

As for AIC and BIC they are criterion based model selection approaches that are typically used for time series and multiple regression analysis, I have not used them extensively. My advice is to understand your data as intimately as you can before allowing a statistical criterion to make choices for you.

If you are doing ANOVAs do means plots. If you are doing regressions do a 2-D or 3-D contour plot.

Hope this helps, Best of luck.

marcioadg
Calcite | Level 5

I appreciate all the input you gave me! That will help me deciding on the model.

Kodml - in that example the variable definitely has biological meaning.

Thanks!

Marcio

SteveDenham
Jade | Level 19

Excluding based on a p value is always dangerous.  What would you be asking if one of the main effects was not significant, but the interaction was significant?  Which makes the results more interpretable?

And, what happens to corrected AIC?  That has been shown to be a better measure of model fit for smaller datasets.

Steve Denham

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

What is ANOVA?

ANOVA, or Analysis Of Variance, is used to compare the averages or means of two or more populations to better understand how they differ. Watch this tutorial for more.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 14 replies
  • 8121 views
  • 7 likes
  • 7 in conversation