Programming the statistical procedures from SAS

Why p=.0393 is not significant at .05 level in GLM?

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 13
Accepted Solution

Why p=.0393 is not significant at .05 level in GLM?

I am using the GLM procedure to compare 3 groups with Tukey's (HSD) test for pairwise comparisons.  Alpha is set to .05.  In Tukey's test, I received one result as: 1 vs 3: (Pr>F) = 0.0393, not marked for significance.  Why is this P not significant at .o5 level?

The 95% Confidence Limits are: -0.5574 and 0.0374, which contains 0,  That is why it was not selected as significant.  But if I use prob, it is supposed to be significant. 

I noticed a similar problem in for another variable.  The Value of P is .0007 and it is not significant at .001 level (99.9%).   (I checked number of zeros many times.)

I think I am seriously missing something here.   Can somebody please explain why such discrepancy?

I have to explain results tomorrow.

Thanks.


Accepted Solutions
Solution
‎04-25-2014 11:36 AM
Respected Advisor
Posts: 2,655

Re: Why p=.0393 is not significant at .05 level in GLM?

I am now confused.  You mentioned specifically (Pr>F) = 0.0393, so I assumed that this was an F value from the omnibus test.  I missed something here, so where was that probability presented ( I don't see it in the excerpt you gave).

I think the key is that the CONTRAST statement in GLM does not allow for adjustments for multiple comparisons.  As a result, you get a p value that is what you would expect if that were the only comparison of interest (the 0.0007).  The adjusted confidence bounds would result in some p value greater than 0.001.

Try adding the following:

lsmeans grpcodes/pdiff adjust=tukey;

This should give a raw p value, equal to that seen in your CONTRAST statement, and an adjusted p value, which would correspond to the confidence bounds from the MEANS statement.

Steve Denham

View solution in original post


All Replies
Respected Advisor
Posts: 2,655

Re: Why p=.0393 is not significant at .05 level in GLM?

When you use multiple comparison adjustments, there are two p values reported.  The first is the raw p, which is like an un-adjusted t-test between the two means/lsmeans.  The second is the adjusted p-value, which is going to be larger.

Also, you may be confusing the omnibus F test p value with any pair-wise comparisons.  The omnibus test does NOT apply a multiplicity correction, as there is only one test.  However, the pairwise comparisons are multiple tests that, in most cases, should be adjusted for multiplicity.

Most of this is explained pretty clearly in the documentation of the ADJUST= option.  However, for a really good and practical explanation I would recommend Peter Westfall's book Multiple Comparisons and Multiple Tests Using SAS, 2nd ed.  See this link:

http://support.sas.com/publishing/authors/westfall.html

Steve Denham

Occasional Contributor
Posts: 13

Re: Why p=.0393 is not significant at .05 level in GLM?

Thanks for the information. But I didn't mention F-value, I gave values from pair-wise comparisons.  I am checking only main effect.  So, I used

proc glm data=&DataName2;

   class grpcode2;

   model BMI = grpcode2 /ss3 ;

   means grpcode2 / tukey alpha=.001 ;  

   contrast 'Obese(1) vs OverWeight(2)' grpcode2 -1 1 0 ;

....

run;

  (I tried with lsmeans/adjust=tukey and had the same results with no confidence limits.)

  The partial results are:

          Difference
                   grpcode2    Between Simultaneous 99.9%
                  Comparison     Means Confidence Limits

                    1 - 2      12.2686   7.9828   16.5543  ***
                    1 - 3      16.6226  12.3368   20.9083  ***
                    2 - 1     -12.2686 -16.5543   -7.9828  ***
                    2 - 3       4.3540  -0.27518.9831
                    3 - 1     -16.6226 -20.9083  -12.3368  ***
                    3 - 2      -4.3540  -8.98310.2751

Contrast                       DF Contrast SS Mean SquareF ValuePr > F

   Obese(1) vs OverWeight(2)       878.020762  878.020762 131.98<.0001
   Obese(1) vs Control(3)          1 1611.807639 1611.807639 242.28<.0001
   OverWeight(2) vs Control(3)     1   94.786580   94.786580  14.250.0007

So, I thought 2 vs 3 should be sig at .001 level.

Occasional Contributor
Posts: 13

Re: Why p=.0393 is not significant at .05 level in GLM?

P-val = 0.0393 from my first Eample and p-val =.0007 from my second Ex (both using MEANS option) are also the same as the adjusted (from second set) p-values obtained using LSMEANS options.

So, are they supposed to be significant  at .05 and .001 levels, respectively, even though CLs don't mark those results for significance in MEANS option?

Solution
‎04-25-2014 11:36 AM
Respected Advisor
Posts: 2,655

Re: Why p=.0393 is not significant at .05 level in GLM?

I am now confused.  You mentioned specifically (Pr>F) = 0.0393, so I assumed that this was an F value from the omnibus test.  I missed something here, so where was that probability presented ( I don't see it in the excerpt you gave).

I think the key is that the CONTRAST statement in GLM does not allow for adjustments for multiple comparisons.  As a result, you get a p value that is what you would expect if that were the only comparison of interest (the 0.0007).  The adjusted confidence bounds would result in some p value greater than 0.001.

Try adding the following:

lsmeans grpcodes/pdiff adjust=tukey;

This should give a raw p value, equal to that seen in your CONTRAST statement, and an adjusted p value, which would correspond to the confidence bounds from the MEANS statement.

Steve Denham

Occasional Contributor
Posts: 13

Re: Why p=.0393 is not significant at .05 level in GLM?

You are right.  I also did study and understood the results correctly.  I switched to  lsmeans option as those results are more clear to me. I used same options as you mentioned in addition to cl just to check CL values.  My problem is now solved.

By the way, (Pr>F) = 0.0393 was from another example in which I had the similar problem.

Thanks for all the explanation.

🔒 This topic is solved and locked.

Need further help from the community? Please ask a new question.

Discussion stats
  • 5 replies
  • 481 views
  • 5 likes
  • 2 in conversation