## Multiple comparison in the unequal variance case using PROC MIXED

Solved
Occasional Contributor
Posts: 7

# Multiple comparison in the unequal variance case using PROC MIXED

I have found procedures such as Tamhanes T2, Dunnets T3 and the Games & Howell procedure that deal with unequal variances in the one-way model. However, I have a Randomized complete block design, which is basically a two-way model. And variances differ quite a lot in the treatment group. In SAS for mixed models 2nd ed. (p. 369) Ramon C. Litell et. al., uses a unequal variance model,

proc mixed data=TV ic;

class age sex;

model time=sex|age/DDFM=KR OUTP=R;

repeated / group=age;

lsmeans age sex / diff adjust=Tukey;

run;

However, I am not sure this is correct since the multiple comparison test (Tukey), uses a pooled estimate for the variance thus affecting p-values when the variances are unequal. Is this the correct way of performing multiple comparison under unequal variance? If yes, how does the Tukey adjustment handle the unequal variances?

Accepted Solutions
Solution
‎04-03-2014 08:40 AM
Posts: 2,655

## Re: Multiple comparison in the unequal variance case using PROC MIXED

Yes.

For your code, it would be:

proc mixed data=TV ic;

class age sex;

model time=sex|age/DDFM=KR OUTP=R;

repeated / group=age;

run;

I picked seed=1 but any value could be inserted.  You do want to specify a seed, so that different runs are identical.

Steve Denham

All Replies
Posts: 2,655

## Re: Multiple comparison in the unequal variance case using PROC MIXED

First point:

The model presented fits a separate variance for each age group, As Westfall et al. in Multiple Comparisons and Multiple Tests Using SAS, 2nd ed. say in Chapter 10, p 274 puts it "However, with extreme heteroscedasticity, Tukey's method can fail miserably."  So, I wouldn't be using method=Tukey in this case.

Second point:

Westfall et al. point out that all methods are approximate under heteroscedastic variances.  In Ch. 10, he gives several methods, and points out the pros and cons of each.  I would suggest the MaxT adjustment.

Steve Denham

Occasional Contributor
Posts: 7

## Re: Multiple comparison in the unequal variance case using PROC MIXED

I look this up! Is the MaxT adjustment implemented in SAS?

Solution
‎04-03-2014 08:40 AM
Posts: 2,655

## Re: Multiple comparison in the unequal variance case using PROC MIXED

Yes.

For your code, it would be:

proc mixed data=TV ic;

class age sex;

model time=sex|age/DDFM=KR OUTP=R;

repeated / group=age;

run;

I picked seed=1 but any value could be inserted.  You do want to specify a seed, so that different runs are identical.

Steve Denham

Occasional Contributor
Posts: 7

## Re: Multiple comparison in the unequal variance case using PROC MIXED

I am still a bit uncertain as to what kind of test is performed under heteroscedasticity and how close to the nominal levels of the presented p-values we could get. Is the method well explained in Multiple Comparisons and Multiple Tests Using SAS, 2nd ed.? I might just pick up a copy to understand this method better!

Posts: 2,655

## Re: Multiple comparison in the unequal variance case using PROC MIXED

Personal opinion:  I think Westfall's book is, or should be, required reading for anyone who works with designed experiments.  There is a lot of theory mixed in, but in a way that makes it easier to understand what the code is doing.  I think the method is well explained, and there are references given that relate to the performance as well as summaries of performance.

Steve Denham

New Contributor
Posts: 2

## Re: Multiple comparison in the unequal variance case using PROC MIXED

Steve,

Would you recommend the sim option as a matter of SOP or are there disadvantages?

Posts: 2,655

## Re: Multiple comparison in the unequal variance case using PROC MIXED

We use it as our standard method. It grew out of the ERROR message you get when Dunnett'-Hsu adjustment (previous standard) failed to converge.  SAS recommends ADJUST=SIMULATE in this case

The biggest disadvantages that we have seen are:

One-Time: an adequate number of simulations to give good results may run into long run times, especially if you are trying to meet an accuracy target, and you have a lot of endpoints.

Two-Because of the behavior of the seed and the RNG stream, it makes BY variable processing much trickier in the sense that getting identical results on separate machines is harder.  We have a good method of macro looping that avoids this problem now, however.

Steve Denham

Occasional Contributor
Posts: 7

## Re: Multiple comparison in the unequal variance case using PROC MIXED

Steve,

I purchased the book and I am well on my way of working through it, it was, just as you said, a really good source when working with designed experiments. Thank you!

I have a one more question that I was hoping you could answer. I am currently working through the examples in the chapter we previously discussed and I was wondering if it would be appropriate to use the maxT/minP method when analysing experiments in a randomized complete block design.

I understand that the subset pivotality condtion is depedent on the hypotheses formed to get a strong control of the FWE. However, in the case where we might suspect heteroscedasticity among group levels of treatment could the single-step maxT or minP be the an appropriate solution? Are there any pitfalls that you are aware of?

Posts: 2,655

## Re: Multiple comparison in the unequal variance case using PROC MIXED

I haven't fallen into any pits yet, but I haven't had severe heteroskedasticity in any of our data.  We do see enough that, as a standard in our mixed model analyses, we model it, and accept the risk of losing power when variances are, in fact, nearly homogeneous.  The maxT, as implemented by method=sim, is our default. Hope this helps.

Steve Denham

Occasional Contributor
Posts: 7

## Re: Multiple comparison in the unequal variance case using PROC MIXED

I will try to find some simulation studies to determine power in these cases or conduct some myself!

New User
Posts: 1

## Re: Multiple comparison in the unequal variance case using PROC MIXED

Here is my code:

proc mixed;

class group;

model MPH = group/ddfm = satterth;

repeated/group = group;

lsmeans group/adjust = tukey diff cl;

The confidence intervals that were computed by SAS did not use Tukey critical values but used t critical values. The output indicates that the Adjustment is Tukey-Kramer so I wonder if there is a bug in the program or an error in my code.

New Contributor
Posts: 4

## Re: Multiple comparison in the unequal variance case using PROC MIXED

Hello Steve,

I was searching the SAS forums for a solution to the same problem.  A question for you regarding your response to the original post:

Is it possible to include an interaction term in the model statement?  I am working with a data set that has two factor variables, SPECIES (three levels) and PARASITISM_STATUS (two levels). The variance in the response variable is much larger in one level of parasitism_status than the other, even when transformed. Equality of variances is not a problem for "species" when the data is log10-transformed. I am interested in the interaction between these two factor variables, so my current model statement is "model y = spp parasitism_status spp*parasitism_status."  Is there a way to include this type of interaction term in the code you provided above?

Many thanks!

Posts: 2,655

## Re: Multiple comparison in the unequal variance case using PROC MIXED

The code I provided used the shorthand expression

sex|age

which is identical to:

sex age sex*age;

So the interaction was included.

For your approach, I would consider using PROC GLIMMIX, doing something like:

proc glimmix data=yourdataset;

class spp parisitism_status;

model y = spp|parasitism_status / link=log; /* Note here y is not transformed prior to analysis */

random _residual_/group=parasitism_status;

lsmeans spp parasitism_status spp*parasitism_status/exp;

lsmestimates <these will compare the lsmeans of interest to address your study objectives, and will adjust for multiple comparisons>;

run;

The lsmestimate statement is one of the finest blades in the Swiss Army knife that is GLIMMIX.  Rather than looking at all possible comparisons, as would the diff option in the lsmeans statement, you can narrow down to those of interest.  This is particularly important when looking at repeated measurements in time, where you really aren't very interested in comparing the mean of group 2 at timepoint 3 to the mean of group 5 at timepoint 11.  By eliminating comparisons such as these, the overall type I error can be controlled better without loss of power.

Steve Denham

New Contributor
Posts: 4

## Re: Multiple comparison in the unequal variance case using PROC MIXED

Thank you so much for your reply, Steve!  And thank you for drawing my attention to the lsmestimate statment in GLIMMIX.  I'm just starting to learn how to use this platform in SAS.

How does the lsmestimate statement differ from the use of contrasts?  Currently I have a GLIMMIX procedure coded for the same dataset I described, and I've used contrast statements to examine specific comparisons. Is this problematic?

☑ This topic is solved.