04-04-2018 02:34 PM
I am using Glimmix to do a variance analysis of 96 treatments, each with 3 blocks (reps). My question is whether tukey would by the correct adjustment to use as this would give 4560 pairwise combos, my lsmeans is shown below and my treatment is "strain"
lsmeans strain/pdiff adjust=tukey;
The experiment is a screen to find the highest NH4 excretion, and I am wondering if another adjustment would make more sense with this number of treatments
04-04-2018 05:29 PM
To me, screening is exploratory. If the objective of this study is to select a smaller number of treatments for future study, I could see doing no adjustment at all. (I'm sure someone somewhere would disagree.)
Or you could explicitly balance the costs of Type I and Type II errors; for example, see Setting an Optimal α That Minimizes Errors in Null Hypothesis Significance Tests; but see LOPSIDED REASONING ON LOPSIDED TESTS AND MULTIPLE COMPARISONS for an opposing opinion.
Of the usual Type I error control methods, I might consider false discovery rate rather than something like Tukey.
I doubt that there is a definitive answer to your question, although there may be something like a consensus in your particular discipline (which might not be the same consensus in another discipline).
04-10-2018 10:52 AM
Thanks for the well thought out reply, I think Tukey increases my type two error rate to a large margin, and having no adjustment changes this.
I am going to inquire with others in my discipline on what they think of having no adjustment for this screen.
Thanks for the advice.
All the best, Chris
04-10-2018 03:17 PM
Using simulation studies, Day and Quinn (1989) COMPARISONS OF TREATMENTS AFTER AN ANALYSIS OF VARIANCE IN ECOLOGY showed that the Tukey method is conservative with respect to Type I error rate; in Fig 1 you can see that the actual Type I error rate is about 0.02 when the nominal rate is 0.05 for their simulation scenario. Consequently, Type II error rate also is excessively increased (i.e., you lose more power). There's no perfect solution