I am using Glimmix to do a variance analysis of 96 treatments, each with 3 blocks (reps). My question is whether tukey would by the correct adjustment to use as this would give 4560 pairwise combos, my lsmeans is shown below and my treatment is "strain"
lsmeans strain/pdiff adjust=tukey;
The experiment is a screen to find the highest NH4 excretion, and I am wondering if another adjustment would make more sense with this number of treatments
Speculating here....
To me, screening is exploratory. If the objective of this study is to select a smaller number of treatments for future study, I could see doing no adjustment at all. (I'm sure someone somewhere would disagree.)
Or you could explicitly balance the costs of Type I and Type II errors; for example, see Setting an Optimal α That Minimizes Errors in Null Hypothesis Significance Tests; but see LOPSIDED REASONING ON LOPSIDED TESTS AND MULTIPLE COMPARISONS for an opposing opinion.
Of the usual Type I error control methods, I might consider false discovery rate rather than something like Tukey.
I doubt that there is a definitive answer to your question, although there may be something like a consensus in your particular discipline (which might not be the same consensus in another discipline).
Thanks for the well thought out reply, I think Tukey increases my type two error rate to a large margin, and having no adjustment changes this.
I am going to inquire with others in my discipline on what they think of having no adjustment for this screen.
Thanks for the advice.
All the best, Chris
Using simulation studies, Day and Quinn (1989) COMPARISONS OF TREATMENTS AFTER AN ANALYSIS OF VARIANCE IN ECOLOGY showed that the Tukey method is conservative with respect to Type I error rate; in Fig 1 you can see that the actual Type I error rate is about 0.02 when the nominal rate is 0.05 for their simulation scenario. Consequently, Type II error rate also is excessively increased (i.e., you lose more power). There's no perfect solution 🙂
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!
ANOVA, or Analysis Of Variance, is used to compare the averages or means of two or more populations to better understand how they differ. Watch this tutorial for more.
Find more tutorials on the SAS Users YouTube channel.