topic Re: LSMean Diffs Sig, Main Effects Not in SAS Procedures

LSMean Diffs Sig, Main Effects Not

MDS — Thu, 08 Oct 2009 18:16:13 GMT

I'm analyzing a clinical study to assess drug effectiveness against several different controls. The study design is rather complicated and involves three factors, cell type (tumor or no tumor), drug treatment (5 levels), and timing of drug treatment (early or late). The design is not fully factorial in that all possible combinations of the three factors were not examined, and I'm guessing that this is the cause of the confusing results that I have gotten.

Specifically, using proc mixed I analyzed the three main factors as well as one interaction of interest (time*treatment), resulting in the following main effects:

Type 3 Tests of Fixed Effects

Num Den
Effect DF DF F Value Pr > F

Cell_Type 1 68 0.16 0.6921
Treatment 4 68 2.07 0.0937
Timing 1 68 27.73 <.0001
Treatment*Timing 3 68 1.39 0.2538

I followed this up with a comparison of lsmean differences to see what the difference between early and late was. Probably as a result of the study design, I got no result for the comparison of interest (see obs 12 below), but in looking at other comparisons, I was surprised to find that several of the treatment and treatment*timing comparisons had p-values <0.05 (see observations 3,6,10,13,15,20,21 below):

Differences of Least Squares Means

Cell _Cell
Obs Effect Type Treatment Timing Type _Treatment _Timing

1 Cell_Type No_Tumor Tumor
2 Treatment DPO Non
3 Treatment DPO PIV
4 Treatment DPO PVI
5 Treatment DPO WN
6 Treatment Non PIV
7 Treatment Non PVI
8 Treatment Non WN
9 Treatment PIV PVI
10 Treatment PIV WN
11 Treatment PVI WN
12 Timing Early Late
13 Treatment*Timing DPO Early DPO Late
14 Treatment*Timing DPO Early Non Early
15 Treatment*Timing DPO Early Non Late
16 Treatment*Timing DPO Early PIV Early
17 Treatment*Timing DPO Early PIV Late
18 Treatment*Timing DPO Early PVI Early
19 Treatment*Timing DPO Early WN Early
20 Treatment*Timing DPO Early WN Late
21 Treatment*Timing DPO Late Non Early

Obs Estimate StdErr DF tValue Probt Alpha Lower Upper

1 -3.6463 9.1696 68 -0.40 0.6921 0.05 -21.9438 14.6513
2 2.6606 6.4839 68 0.41 0.6828 0.05 -10.2777 15.5990
3 17.0260 6.5986 68 2.58 0.0120 0.05 3.8586 30.1934
4 . . . . . . . .
5 1.9819 6.4839 68 0.31 0.7608 0.05 -10.9565 14.9202
6 14.3654 6.5986 68 2.18 0.0330 0.05 1.1980 27.5327
7 . . . . . . . .
8 -0.6788 6.4839 68 -0.10 0.9169 0.05 -13.6171 12.2596
9 . . . . . . . .
10 -15.0441 6.5986 68 -2.28 0.0258 0.05 -28.2115 -1.8767
11 . . . . . . . .
12 . . . . . . . .
13 -35.3287 9.1696 68 -3.85 0.0003 0.05 -53.6263 -17.0312
14 -0.4025 9.1696 68 -0.04 0.9651 0.05 -18.7001 17.8951
15 -29.6050 9.1696 68 -3.23 0.0019 0.05 -47.9026 -11.3074
16 4.1182 9.4914 68 0.43 0.6657 0.05 -14.8216 23.0580
17 -5.3950 9.1696 68 -0.59 0.5582 0.05 -23.6926 12.9026
18 -0.7246 9.4914 68 -0.08 0.9394 0.05 -19.6645 18.2152
19 -3.9863 9.1696 68 -0.43 0.6651 0.05 -22.2838 14.3113
20 -27.3787 9.1696 68 -2.99 0.0039 0.05 -45.6763 -9.0812
21 34.9263 9.1696 68 3.81 0.0003 0.05 16.6287 53.2238

This seems wrong. If the main effect does not have a p-value<0.05, then how could the pairwise comparisons have p-values<0.05? I would appreciate some explanation and or a reference to an appropriate source of information.

Thanks in advance,
Candan

Re: LSMean Diffs Sig, Main Effects Not

Paige — Fri, 09 Oct 2009 13:46:09 GMT

The main effect tests different things than the pairwise effect. The main effect tests if the n levels show a statistically significant difference; it uses the variability of the means of the n levels. The pairwise test compares two levels; it uses the variability of the means of the 2 levels.

This has nothing to do with what type of design you have (although the type of design could affect the tests). It can be true for any design.

In the future, when discussing designed experiments, it would help if you told us what design you do have, rather than what design you don't have.

Re: LSMean Diffs Sig, Main Effects Not

Doc_Duke — Fri, 09 Oct 2009 18:01:22 GMT

MDS,

In addition to Paige's comments, I'll add a caution. You are getting into the land of multiple comparisons and it can be a slippery slope to a "fishing expedition" and declaring a false positive conclusion.

Frank Harrell, in his book on regression methods, has a good description of how to develop
modeling strategies to control the Type I error rate in a reasonable way.

Doc Muhlbaier
Duke

Re: LSMean Diffs Sig, Main Effects Not

palolix — Mon, 11 Mar 2013 17:22:29 GMT

So in such situation, should you use the adjust=bon or tukey statement because the are very conservative, so you should probably get no significant differences between levels for a main effect which is not significant ??

Caroline Eberlein

Re: LSMean Diffs Sig, Main Effects Not

SteveDenham — Tue, 12 Mar 2013 12:09:45 GMT

Even with very conservative adjustment methods, you could still get pairwise "significant" differences. That is why many texts recommend that pairwise comparisons only be carried out when the "overall" test is significant.

Steve Denham