topic Re: What adjustment to pick in a GLIMMIX variance analysis in Statistical Procedures

What adjustment to pick in a GLIMMIX variance analysis

cdumigan — Wed, 04 Apr 2018 18:34:54 GMT

I am using Glimmix to do a variance analysis of 96 treatments, each with 3 blocks (reps). My question is whether tukey would by the correct adjustment to use as this would give 4560 pairwise combos, my lsmeans is shown below and my treatment is "strain"

lsmeans strain/pdiff adjust=tukey;

The experiment is a screen to find the highest NH4 excretion, and I am wondering if another adjustment would make more sense with this number of treatments

Re: What adjustment to pick in a GLIMMIX variance analysis

sld — Wed, 04 Apr 2018 21:29:27 GMT

Speculating here....

To me, screening is exploratory. If the objective of this study is to select a smaller number of treatments for future study, I could see doing no adjustment at all. (I'm sure someone somewhere would disagree.)

Or you could explicitly balance the costs of Type I and Type II errors; for example, see Setting an Optimal α That Minimizes Errors in Null Hypothesis Significance Tests; but see LOPSIDED REASONING ON LOPSIDED TESTS AND MULTIPLE COMPARISONS for an opposing opinion.

Of the usual Type I error control methods, I might consider false discovery rate rather than something like Tukey.

I doubt that there is a definitive answer to your question, although there may be something like a consensus in your particular discipline (which might not be the same consensus in another discipline).

Re: What adjustment to pick in a GLIMMIX variance analysis

cdumigan — Tue, 10 Apr 2018 14:52:26 GMT

Thanks for the well thought out reply, I think Tukey increases my type two error rate to a large margin, and having no adjustment changes this.

I am going to inquire with others in my discipline on what they think of having no adjustment for this screen.

Thanks for the advice.

All the best, Chris

Re: What adjustment to pick in a GLIMMIX variance analysis

sld — Tue, 10 Apr 2018 19:17:06 GMT

Using simulation studies, Day and Quinn (1989) COMPARISONS OF TREATMENTS AFTER AN ANALYSIS OF VARIANCE IN ECOLOGY showed that the Tukey method is conservative with respect to Type I error rate; in Fig 1 you can see that the actual Type I error rate is about 0.02 when the nominal rate is 0.05 for their simulation scenario. Consequently, Type II error rate also is excessively increased (i.e., you lose more power). There's no perfect solution 🙂