turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Find a Community

- Home
- /
- Analytics
- /
- Stat Procs
- /
- Why p=.0393 is not significant at .05 level in GLM...

Topic Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

04-24-2014 10:43 AM

I am using the GLM procedure to compare 3 groups with Tukey's (HSD) test for pairwise comparisons. Alpha is set to .05. In Tukey's test, I received one result as: 1 vs 3: (Pr>F) = 0.0393, not marked for significance. Why is this P not significant at .o5 level?

The 95% Confidence Limits are: -0.5574 and 0.0374, which contains 0, That is why it was not selected as significant. But if I use prob, it is supposed to be significant.

I noticed a similar problem in for another variable. The Value of P is .0007 and it is not significant at .001 level (99.9%). (I checked number of zeros many times.)

I think I am seriously missing something here. Can somebody please explain why such discrepancy?

I have to explain results tomorrow.

Thanks.

Accepted Solutions

Solution

04-25-2014
11:36 AM

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

04-25-2014 11:36 AM

I am now confused. You mentioned specifically (Pr>F) = 0.0393, so I assumed that this was an F value from the omnibus test. I missed something here, so where was that probability presented ( I don't see it in the excerpt you gave).

I think the key is that the CONTRAST statement in GLM does not allow for adjustments for multiple comparisons. As a result, you get a p value that is what you would expect if that were the only comparison of interest (the 0.0007). The adjusted confidence bounds would result in some p value greater than 0.001.

Try adding the following:

lsmeans grpcodes/pdiff adjust=tukey;

This should give a raw p value, equal to that seen in your CONTRAST statement, and an adjusted p value, which would correspond to the confidence bounds from the MEANS statement.

Steve Denham

All Replies

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

04-24-2014 10:53 AM

When you use multiple comparison adjustments, there are two p values reported. The first is the raw p, which is like an un-adjusted t-test between the two means/lsmeans. The second is the adjusted p-value, which is going to be larger.

Also, you may be confusing the omnibus F test p value with any pair-wise comparisons. The omnibus test does NOT apply a multiplicity correction, as there is only one test. However, the pairwise comparisons are multiple tests that, in most cases, should be adjusted for multiplicity.

Most of this is explained pretty clearly in the documentation of the ADJUST= option. However, for a really good and practical explanation I would recommend Peter Westfall's book *Multiple Comparisons and Multiple Tests Using SAS, 2nd ed.* See this link:

http://support.sas.com/publishing/authors/westfall.html

Steve Denham

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

04-24-2014 11:47 AM

Thanks for the information. But I didn't mention F-value, I gave values from pair-wise comparisons. I am checking only main effect. So, I used

proc glm data=&DataName2;

class grpcode2;

model BMI = grpcode2 /ss3 ;

means grpcode2 / tukey alpha=.001 ;

contrast 'Obese(1) vs OverWeight(2)' grpcode2 -1 1 0 ;

....

run;

(I tried with lsmeans/adjust=tukey and had the same results with no confidence limits.)

The partial results are:

Difference | |||

grpcode2 | Between | Simultaneous 99.9% | |

Comparison | Means | Confidence Limits |

1 - 2 | 12.2686 | 7.9828 16.5543 *** | ||

1 - 3 | 16.6226 | 12.3368 20.9083 *** | ||

2 - 1 | -12.2686 | -16.5543 -7.9828 *** | ||

2 - 3 | 4.3540 | -0.2751 | 8.9831 | |

3 - 1 | -16.6226 | -20.9083 -12.3368 *** | ||

3 - 2 | -4.3540 | -8.9831 | 0.2751 |

Contrast | DF | Contrast SS | Mean Square | F Value | Pr > F |

Obese(1) vs OverWeight(2) | 1 | 878.020762 | 878.020762 | 131.98 | <.0001 |

Obese(1) vs Control(3) | 1 | 1611.807639 | 1611.807639 | 242.28 | <.0001 |

OverWeight(2) vs Control(3) | 1 | 94.786580 | 94.786580 | 14.25 | 0.0007 |

So, I thought 2 vs 3 should be sig at .001 level.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

04-24-2014 12:57 PM

P-val = 0.0393 from my first Eample and p-val =.0007 from my second Ex (both using MEANS option) are also the same as the **adjusted (from second set)** p-values obtained using LSMEANS options.

So, are they supposed to be significant at .05 and .001 levels, respectively, even though CLs don't mark those results for significance in MEANS option?

Solution

04-25-2014
11:36 AM

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

04-25-2014 11:36 AM

I am now confused. You mentioned specifically (Pr>F) = 0.0393, so I assumed that this was an F value from the omnibus test. I missed something here, so where was that probability presented ( I don't see it in the excerpt you gave).

I think the key is that the CONTRAST statement in GLM does not allow for adjustments for multiple comparisons. As a result, you get a p value that is what you would expect if that were the only comparison of interest (the 0.0007). The adjusted confidence bounds would result in some p value greater than 0.001.

Try adding the following:

lsmeans grpcodes/pdiff adjust=tukey;

This should give a raw p value, equal to that seen in your CONTRAST statement, and an adjusted p value, which would correspond to the confidence bounds from the MEANS statement.

Steve Denham

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Email to a Friend
- Report Inappropriate Content

04-25-2014 11:56 AM

You are right. I also did study and understood the results correctly. I switched to lsmeans option as those results are more clear to me. I used same options as you mentioned in addition to cl just to check CL values. My problem is now solved.

By the way, (Pr>F) = 0.0393 was from another example in which I had the similar problem.

Thanks for all the explanation.