topic Re: Three-Way ANOVA and Adjusted Tukey Comparisons in Statistical Procedures

Three-Way ANOVA and Adjusted Tukey Comparisons

ANKH1 — Tue, 21 Apr 2020 17:59:38 GMT

Hi,

In a 3-Way ANOVA analyses, is it possible to decide a priori what comparisons between different factors we want to look at? We would like to look at tukey's where there is a two factor interaction. Is this valid since tukey's uses adjusted p values per tested comparison?

This is the code for the overall 3 Way ANOVA (2*2*2):

PROC GLIMMIX DATA= data;
CLASS id dose form gen;
MODEL var= dose|form|gen;
RANDOM residual/ subject = ID type=CS;
LSMEANS dose|form|gen /ADJUST=TUKEY;
RUN;QUIT;

These are the results for type 3 effects

This is the output where a two factor interaction was found:

Re: Three-Way ANOVA and Adjusted Tukey Comparisons

SteveDenham — Tue, 21 Apr 2020 19:19:10 GMT

It is possible to make those a priori comparisons, and then use Tukey's adjustment on them. Welcome to the LSMESTIMATE statement. You can (carefully) plug in what you want to compare (say, level 1 of dose to level 2 of dose at level 1 of form AND at level 2 of form) with something like:

lsmestimate dose*form 'level 1 of dose v level 2 of dose at level 1 of form' 1 -1 0 0,
                      'level 1 of dose v level 2 of dose at level 12of form' 0 0 1 -1/adjust=tukey;

Based on your F tests, that is probably not one of the comparisons you might be interested in, but it should give you the idea of how to set up what you are interested in.

SteveDenham

Re: Three-Way ANOVA and Adjusted Tukey Comparisons

SteveDenham — Tue, 21 Apr 2020 19:28:52 GMT

Shoot. i have a typo in the box. It should read:

lsmestimate dose*form 'level 1 of dose v level 2 of dose at level 1 of form' 1 -1 0 0,
                      'level 1 of dose v level 2 of dose at level 2 of form' 0 0 1 -1/adjust=tukey;

The snippet of output that you showed reveals that Tukey's is adjusting for all possible comparisons (and I assume that 2 of them were cut off, as there should be 6), whereas this approach only looks at the simple effect of dose at each level of form.

Re: Three-Way ANOVA and Adjusted Tukey Comparisons

ANKH1 — Tue, 21 Apr 2020 19:48:31 GMT

Thank you! Sorry about the two missing lines for tukey's. What if I want to look at all pairwise comparisons between lsmeans calculated from the output?

Is it valid to look at all 6 pairwise comparisons coming from tukey-kramer from my previous output?

Thanks again!

Re: Three-Way ANOVA and Adjusted Tukey Comparisons

SteveDenham — Tue, 21 Apr 2020 19:57:11 GMT

Well, you can do that, but be aware that the way you have it set up right now, it adjusts for all 105 pairwise comparisons, including main effects, 2 way interactions and 3 way interactions (hope my combinatorics calculations are correct). Are you sure that all of those are of interest? If so, then the T-K adjusted p values (last column of the Differences table in the output) are the ones you want.

SteveDenham

Re: Three-Way ANOVA and Adjusted Tukey Comparisons

ANKH1 — Tue, 21 Apr 2020 20:00:14 GMT

By choosing to look at Adj T K values adjusted for all 105 comparisons am I increasing my type 1 error?

Re: Three-Way ANOVA and Adjusted Tukey Comparisons

sld — Wed, 22 Apr 2020 02:42:07 GMT

@SteveDenham makes the perfect point.

In my opinion, once you move past Tukey-adjusted comparisons of main effect means, then you need to think about how to interpret interaction in your particular scenario and what comparisons (i.e., research hypotheses) are appropriate. Interaction is, fundamentally, comparisons of pairwise comparisons. Except in the simplest of cases, pairwise comparisons among interaction means will not give you the insight you need for higher order interactions. And if you use Tukey adjusted pairwise comparisons, you are controlling for way more comparisons that you intend (that was Steve's point).

I never apply Tukey to all pairwise comparisons of interaction means. That approach is answering the wrong question. I think about my research question (hopefully I've done that BEFORE I collected my data), and then I pose hypotheses that I can express as contrasts that I can test.

Yes, it looks subjective if you have not done the work a priori, and you do have to be careful, but if you've done your thinking a priori, then...cool (if uncommon).

Re: Three-Way ANOVA and Adjusted Tukey Comparisons

sld — Wed, 22 Apr 2020 02:44:30 GMT

No, you are not increasing your Type I error. But you are seriously decreasing your power (i.e., you are increasing your Type II error). And you are not really addressing any hypotheses of interest.

Re: Three-Way ANOVA and Adjusted Tukey Comparisons

SteveDenham — Wed, 22 Apr 2020 10:36:36 GMT

Thanks @sld . I do want to correct one thing - my combinatorics calculation was off. The correct number should be 49 rather than 105;

Main effects comparisons: 3 (3 times (2take2))

Two way effects comparisons: 18 (3 times (4take2))

Three way effects comparisons: 28 (8take2)

So, if you adjust in the lsmeans statement with your current syntax, you overadjust and demolish your ability to control the balance between type 1 and type 2 errors.

LSMESTIMATE enables you to specify the contrasts for the hypotheses of interest, and then make multiplicity adjustments as needed. That "as needed" has spawned an entire cottage industry of multiplicty adjustment methods, papers and arguments. I could make the case that if you specify the comparisons a priori, you should not adjust. I can also make a case for not paying much attention to the p values themselves, but looking at the confidence intervals around the estimated differences. In the end, you have to think long and hard about what questions you need statistical support for.

SteveDenham

Re: Three-Way ANOVA and Adjusted Tukey Comparisons

ANKH1 — Wed, 22 Apr 2020 14:21:47 GMT

Your posts have been really helpful and I've learned a lot. When using LSMESTIMATE, can I have one contrast for a two way interaction and one contrast per main effect? Also, do you recommend ADJUST= BON for pairwise comparisons using LSMESTIMATE? Apparently Tukey is not an option.

Thanks again!

Re: Three-Way ANOVA and Adjusted Tukey Comparisons

SteveDenham — Wed, 22 Apr 2020 18:41:35 GMT

The Bonferroni correction is needlessly conservative. Once you get this many comparisons, you can lose a lot of power to detect real differences if you use Bonferroni. I happen to like this reference

Edwards, D., and Berry, J. J. (1987). “The Efficiency of Simulation-Based Multiple Comparisons.” Biometrics 43:913–928

This method is called by ADJUST=SIMULATE. Be sure to read up on the options for this, especially the SEED= option, so that you can get reproducible results on different machines or at different times.

SteveDenham

Re: Three-Way ANOVA and Adjusted Tukey Comparisons

ANKH1 — Thu, 23 Apr 2020 14:22:30 GMT

This is great! I will read this paper. Another last question, with LSESTIMATE I can decide a priori what 2 way comparisons and main effects I am interested in, correct?

Thanks again for your help, it is very much appreciated.

Re: Three-Way ANOVA and Adjusted Tukey Comparisons

SteveDenham — Fri, 24 Apr 2020 11:32:57 GMT

That is correct.

SteveDenham

Re: Three-Way ANOVA and Adjusted Tukey Comparisons

ANKH1 — Mon, 27 Apr 2020 13:43:09 GMT

Thank you for all your help!