How to get a pooled Tukey-Kramer adjusted p-value after multiple imputation

[GiaLee]

I'm running a mixed model and using multiple imputation to handle missing covariates.

I used the `lsmeans test/pdiff adjust=Tukey` statement to compare each group.

However, after `PROC MIANALYZE`, it only pools the raw p-values.

I searched various forums and noticed that `PROC MIANALYZE` cannot generate pooled adjusted p-values. I also saw suggestions to use `PROC MULTTEST`, but it doesn't support the Tukey-Kramer adjustment.

Does anyone know how I could obtain the pooled adjusted p-value? Or how could I calculate the pooled adjusted p-value using the pooled raw p-value? 

Thanks!

[SteveDenham]

I make no promises that this will work, but consider the adjusted p values to be random draws from a uniform distribution, bounded above by the largest observed adjusted p value for a fixed pair being compared and below by the smallest observed p value for that pair. That gives a standard deviation across the imputations of SD = (max - min)/sqrt(12), from which you could calculate STDERR = SD/sqrt(N), where N is the number of imputations. Put the adjusted p values and associated standard errors into a data set of estimates and stderr of estimates. Then this might give interesting results:

proc mianalyze data=<your data with adj p values and approx std errors edf=<residual degrees of freedom?>;
   modeleffects adj_p;
   stderr se_ad_pj;
run;

You might have to put out a call for @SAS_Rob or some of the other posters who use MI and MIANALYZE a lot more than me. I was just trying to come up with a standard error estimate for the p values. I know that the possible range for the p values is [0, 1], so it may be that a more correct estimate for the standard error would be 1/(sqrt(12)*sqrt(N)) for every observation. However if that is the case than the value across all of the adjusted p values for a given pair of means is simply the arithmetic mean, as each observation would have an identical weight.

SteveDenham

[GiaLee]

Thanks for your answer! This method looks good, but it is a little bit complex for me since I'm not very experienced with statistical rationale. I was thinking of changing the method to Sidak (more powerful than Bonferroni) so I could use Proc Multtest. Thank you again!

Gia

[SAS_Rob]

I suggest that you combine the LsMeans and standard errors from MIXED in MIANALYZE.  You could then save the p-values from the combined estimates to a SAS data set and feed them into MULTTEST.  You would have to use a different adjustment method than Tukey like Bonferroni since MULTTEST will not compute Tukey p-values from a data set of raw p-values.

[GiaLee]

Thank you, Rob.

I changed the method to adjust by Sidak.

I have obtained estimates, StdErr, DF, tValue, Probt, lower, and Upper from the Proc MIANALYZE.

I found relative code from the PROC MULTTEST documentation,  and adjusted the code to fit my data.

Q1. May I know if the below method aligns with what you mentioned? Input the raw p-value and adjust them directly?

Q2. How could I obtain the adjusted Lower and adjusted Upper then?

Thank you!

data test;
   input Comparison $ Raw_P @@;
   datalines;
12vs18  0.0097     
12vs24  0.09
12vs32  0.01
12vs40 0.0097 
12vs6   0.09
18vs24 0.01
18vs32 0.008
18vs40 0.2
18vs6 0.01
24vs32 0.009
24vs40 0.08
24vs6 0.07
32vs40 0.008
32vs6 0.09
40vs6 0.01
;

proc multtest inpvalues=test sidak;
run;

[SteveDenham]

I still don't see how this is going to give you a "pooled" p value. It should give you 15 adjusted p values. I might have misinterpreted the request but I thought you were looking for a single pooled p value.

SteveDenham

[GiaLee]

Thanks for your reply. I may not have described the question clearly. I used LSmeans in the Proc Mixed to compare between each class, and each comparison has a p-value. And I would like to get the pooled p-value for each of these comparisons.
I found this discussion very useful, as it helped us obtain the pooled p-value and CI. However, it cannot generate the pooled adjusted p-value directly)
https://communities.sas.com/t5/Statistical-Procedures/combine-the-LSMEANS-and-differences-in-LSMEANS...

[SAS_Rob · Accepted solution]

I assume your input data set in this case contains the p-values for each of the comparisons from Proc MIANALYZE.  If that is the case, then, yes, this is what I had in mind.

As far as adjusted CL, I don't really have a good suggestion since MULTTEST only deals with p-values.  The only obvious idea that comes to mind would be to do a Bonferroni-type adjustment by adjusting the value of the ALPHA= option in Proc MIANALYZE such that it is alpha/m where m is the number of comparisons.  There may be others that you could manually program in a data step using the output from MIANALYZE, but that would take some programming on your part.

[Season]

I am also currently studying multiple comparison procedures. I did not know that PROC MULTTEST can plug-in raw P-values before. That is a very useful piece of information.

---

@SAS_Rob wrote:

You would have to use a different adjustment method than Tukey like Bonferroni since MULTTEST will not compute Tukey p-values from a data set of raw p-values.

---

Bonferroni methods are conservative. Are there any macros capable of plugging in raw P-values to implement Tukey's method?

Thanks!