09-17-2014 12:01 AM
I have a question about SAS's implementation of Scheffe's adjustment for multiple comparisons. My understanding of Scheffe (and indeed looking at the formula) is that it accounts for all possible contrasts, and therefore does not depend on the number of contrasts being performed (unlike, say, Bonferroni). This means that regardless of how many other contrasts I'm performing, my adjusted p-value should stay the same. I'm finding that not to be quite the case.
Attached is a SAS dataset and GLIMMIX code, where I perform several LSMEstimate statements with a decreasing number of contrasts. At first, as expected, the Scheffe-adjusted p-values do not change from 7 contrasts down to 4. However, at 3, the p-values change, and at 1 they change again. I tried to change these bottom few to bonferroni or sidak, thinking that perhaps SAS calculates when one of those would be better for the data, but those p-values don't match. I'm a little confused why this is happening. It somewhat makes sense that at only 1 contrast, adjust = scheffe would not be used, since there aren't multiple comparisons, but that would be misleading if, for example, I wanted one extra contrast in addition to the pairwise ones from LSMeans. Any one know?