Why p=.0393 is not significant at .05 level in GLM?

[sasuser3]

I am using the GLM procedure to compare 3 groups with Tukey's (HSD) test for pairwise comparisons.  Alpha is set to .05.  In Tukey's test, I received one result as: 1 vs 3: (Pr>F) = 0.0393, not marked for significance.  Why is this P not significant at .o5 level?

The 95% Confidence Limits are: -0.5574 and 0.0374, which contains 0,  That is why it was not selected as significant.  But if I use prob, it is supposed to be significant. 

I noticed a similar problem in for another variable.  The Value of P is .0007 and it is not significant at .001 level (99.9%).   (I checked number of zeros many times.)

I think I am seriously missing something here.   Can somebody please explain why such discrepancy?

I have to explain results tomorrow.

Thanks.

[SteveDenham]

When you use multiple comparison adjustments, there are two p values reported.  The first is the raw p, which is like an un-adjusted t-test between the two means/lsmeans.  The second is the adjusted p-value, which is going to be larger.

Also, you may be confusing the omnibus F test p value with any pair-wise comparisons.  The omnibus test does NOT apply a multiplicity correction, as there is only one test.  However, the pairwise comparisons are multiple tests that, in most cases, should be adjusted for multiplicity.

Most of this is explained pretty clearly in the documentation of the ADJUST= option.  However, for a really good and practical explanation I would recommend Peter Westfall's book Multiple Comparisons and Multiple Tests Using SAS, 2nd ed.  See this link:

http://support.sas.com/publishing/authors/westfall.html

Steve Denham

[sasuser3]

Thanks for the information. But I didn't mention F-value, I gave values from pair-wise comparisons.  I am checking only main effect.  So, I used

proc glm data=&DataName2;

   class grpcode2;

   model BMI = grpcode2 /ss3 ;

   means grpcode2 / tukey alpha=.001 ;  

   contrast 'Obese(1) vs OverWeight(2)' grpcode2 -1 1 0 ;

....

run;

  (I tried with lsmeans/adjust=tukey and had the same results with no confidence limits.)

  The partial results are:

	Difference
	grpcode2	Between	Simultaneous 99.9%
	Comparison	Means	Confidence Limits

	1 - 2	12.2686	7.9828   16.5543  ***
	1 - 3	16.6226	12.3368   20.9083  ***
	2 - 1	-12.2686	-16.5543   -7.9828  ***
	2 - 3	4.3540	-0.2751	8.9831
	3 - 1	-16.6226	-20.9083  -12.3368  ***
	3 - 2	-4.3540	-8.9831	0.2751

Contrast

DF

Contrast SS

Mean Square

F Value

Pr > F

Obese(1) vs OverWeight(2)	1	878.020762	878.020762	131.98	<.0001
Obese(1) vs Control(3)	1	1611.807639	1611.807639	242.28	<.0001
OverWeight(2) vs Control(3)	1	94.786580	94.786580	14.25	0.0007

So, I thought 2 vs 3 should be sig at .001 level.

[sasuser3]

P-val = 0.0393 from my first Eample and p-val =.0007 from my second Ex (both using MEANS option) are also the same as the adjusted (from second set) p-values obtained using LSMEANS options.

So, are they supposed to be significant  at .05 and .001 levels, respectively, even though CLs don't mark those results for significance in MEANS option?

[SteveDenham · Accepted solution]

I am now confused.  You mentioned specifically (Pr>F) = 0.0393, so I assumed that this was an F value from the omnibus test.  I missed something here, so where was that probability presented ( I don't see it in the excerpt you gave).

I think the key is that the CONTRAST statement in GLM does not allow for adjustments for multiple comparisons.  As a result, you get a p value that is what you would expect if that were the only comparison of interest (the 0.0007).  The adjusted confidence bounds would result in some p value greater than 0.001.

Try adding the following:

lsmeans grpcodes/pdiff adjust=tukey;

This should give a raw p value, equal to that seen in your CONTRAST statement, and an adjusted p value, which would correspond to the confidence bounds from the MEANS statement.

Steve Denham

[sasuser3]

You are right.  I also did study and understood the results correctly.  I switched to  lsmeans option as those results are more clear to me. I used same options as you mentioned in addition to cl just to check CL values.  My problem is now solved.

By the way, (Pr>F) = 0.0393 was from another example in which I had the similar problem.

Thanks for all the explanation.