Programming the statistical procedures from SAS

Multiple comparison adjusted and unadjusted p-values are the same

Accepted Solution Solved
Reply
Contributor
Posts: 26
Accepted Solution

Multiple comparison adjusted and unadjusted p-values are the same

When I run a repeated measures analysis using PROC GLIMMIX, no matter which method of multiple comparison adjustment I use, the unadjusted and adjusted p-values come out the same.  

 

I can't decipher if I'm misunderstanding the statistics and/or coding, or if maybe the adjustments are so minor that they go past 4 digits and aren't displayed.

 

Example of the code:

 

proc glimmix data=all; 
class id epoch day; 
model weight = epoch|day weight0 /solution ddfm=kr;
random day/ sub=id type=cs residual;
lsmeans epoch*day / slicediff=day adjust=bon stepdown adjdfe=row;
run; 

 

Output:

              Simple Effect Comparisons of Epoch*day Least Squares Means By day
                             Adjustment for Multiple Comparisons: Holm

    Simple
    Effect                                  Standard
    Level     Epoch    Epoch    Estimate       Error       DF    t Value    Pr > |t|     Adj P

    day 1     1        2         -175.07      136.93    104.9      -1.28      0.2039    0.2039
    day 2     1        2         15.4559      136.93    104.9       0.11      0.9103    0.9103
    day 3     1        2          148.66      136.93    104.9       1.09      0.2801    0.2801
    day 4     1        2          258.16      137.61    106.6       1.88      0.0634    0.0634
    day 5     1        2          303.35      139.06    110.3       2.18      0.0313    0.0313
    day 6     1        2          426.31      146.71    130.6       2.91      0.0043    0.0043
    day 7     1        2          369.38      152.58    146.9       2.42      0.0167    0.0167

Any insights would be appreciated.

Thanks.


Accepted Solutions
Solution
‎01-16-2017 01:51 PM
Frequent Contributor
Frequent Contributor
Posts: 133

Re: Multiple comparison adjusted and unadjusted p-values are the same

With SLICEDIFF, the adjustment for multiple comparisons is computed within the slice (here, DAY). Documentation is your friend. For the SLICEDIFF option in the LSMEANS statement in GLIMMIX you will find:

 

"When the ADJUST= option is specified, the GLIMMIX procedure also adjusts the tests for multiplicity. The adjustment is based on the number of comparisons within each level of the SLICEDIFF= effect..."

 

EPOCH has only two levels, so there is only one pairwise comparison, and so there is no need to adjust for multiple comparisons and no effect of adjustment.

 

View solution in original post


All Replies
Grand Advisor
Posts: 10,062

Re: Multiple comparison adjusted and unadjusted p-values are the same

Without data or at least the procedure's other information we could well be guessing things that have no bearing on your situation.

 

If you suspect very small differences then send the results to an output data set using the ODS OUTPUT statement and then print or use other display tool with more digits in the display instead of the default p-value format.

 

If the data isn't sensitive you could use the instructions here: https://communities.sas.com/t5/SAS-Communities-Library/How-to-create-a-data-step-version-of-your-dat... to create data step code that would allow us to duplicate results and look for other issues. Subset the data to only the model variables.

 

Or create another data set without sensitive data that replicates the behavior and post that.

Solution
‎01-16-2017 01:51 PM
Frequent Contributor
Frequent Contributor
Posts: 133

Re: Multiple comparison adjusted and unadjusted p-values are the same

With SLICEDIFF, the adjustment for multiple comparisons is computed within the slice (here, DAY). Documentation is your friend. For the SLICEDIFF option in the LSMEANS statement in GLIMMIX you will find:

 

"When the ADJUST= option is specified, the GLIMMIX procedure also adjusts the tests for multiplicity. The adjustment is based on the number of comparisons within each level of the SLICEDIFF= effect..."

 

EPOCH has only two levels, so there is only one pairwise comparison, and so there is no need to adjust for multiple comparisons and no effect of adjustment.

 

Contributor
Posts: 26

Re: Multiple comparison adjusted and unadjusted p-values are the same

Great thanks, that makes sense.  I read through the documentation, but I figured I must be overlooking something.

So then do I understand that there is no built-in option for doing a multiplicity adjustment across the 7 days without writing specific estimate/lsmestimate statements?

Frequent Contributor
Frequent Contributor
Posts: 133

Re: Multiple comparison adjusted and unadjusted p-values are the same

Not that I know of.

 

I think something like this would work:

 

    lsmestimate epoch*day   "Day 1" [1, 1 1] [-1, 2 1],
                            "Day 2" [1, 1 2] [-1, 2 2],
                            "Day 3" [1, 1 3] [-1, 2 3],
                            "Day 4" [1, 1 4] [-1, 2 4],
                            "Day 5" [1, 1 5] [-1, 2 5],
                            "Day 6" [1, 1 6] [-1, 2 6],
                            "Day 7" [1, 1 7] [-1, 2 7]
    /  at means adjust=bon stepdown adjdfe=row;

You could use AT=<value> to specify your desired value of WEIGHT0.

 

Or you can capture the lsmestimates in a SAS dataset using ODS OUTPUT, do any necessary pre-processing, and then use that dataset in the MULTTEST procedure; see

 

https://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_multtest_se...

Contributor
Posts: 26

Re: Multiple comparison adjusted and unadjusted p-values are the same

Fantastic. Thanks, that's very helpful.
Respected Advisor
Posts: 2,655

Re: Multiple comparison adjusted and unadjusted p-values are the same

This is exactly the approach we use in analyzing repeated measures in tox experiments.  Hats off to @sld.

 

Believe me, it was a bit of a shock that SLICEDIFF didn't adjust the way we naively thought it would.  And there the answer was in the documentation.  We now rely on LSMESTIMATE statements almost exclusively.

 

Steve Denham

☑ This topic is SOLVED.

Need further help from the community? Please ask a new question.

Discussion stats
  • 6 replies
  • 229 views
  • 7 likes
  • 4 in conversation