Adjusted pvalue with Proc Mianalyze

[Clemence]

Hello,

I use an multiple imputation and I realize two tests, so I adjusted the pvalue with the bonferonni method. But, how can I have the adjusted p-value with PROC MIANALYZE ? 

Here my code : 

proc mixed data=v1_v2_im method=reml;
class PRDT (ref="Placebo") TV_VISIT_TMP(ref="V1") SUBJID;
model changeVn_V0 = PRDT TV_VISIT_TMP PRDT*TV_VISIT_TMP / s ddfm=kenwardroger;
repeated TV_VISIT_TMP / subject=SUBJID type=CS;
lsmeans PRDT*TV_VISIT_TMP / slice=TV_VISIT_TMP cl diff adjust=bon;
slice PRDT*TV_VISIT_TMP / sliceby(TV_VISIT_TMP="V1") cl pdiff=control("Placebo" "V1");
slice PRDT*TV_VISIT_TMP / sliceby(TV_VISIT_TMP="V2") cl pdiff=control("Placebo" "V2");
by _Imputation_;

ods output SolutionF=mixparms;
ods output SliceDiffs=slice;
ods output Slices=slice2;
ods output Diffs=difs;
run;

It's in the table "Difs" that there are pvalues adjusted. But, I don't know how to use PROC MIANALYZE to pooling adjusted pvalues ?

Thanks for your answer.

Clémence 

[SAS_Rob]

Proc MIANALYZE does not offer any kind of p-value adjustments.  You could save the unadjusted p-values from MIANALYZE to a SAS data set and then feed those p-values into Proc MULTTEST to get adjusted p-values.

[Clemence]

Ok, thanks for your answer. And after use PROC MULTEST, I can use PROC MIANALYZE to pooling ?

Clemence

[Season]

You mentioned a very important limitation of PROC MIANALYZE that seems to have been ignored by the majority of literature introducing this module.

To the best of my knowledge, currently existent multiple comparison procedures adjust P-values on a single dataset. I have personally searched for guidelines on how multiple comparisons should be done when it is complicated by multiple imputation, which generates multiple imputed datasets. Unfortunately, I have not retrieved any guideline on that.

Still, a heuristic approach to this issue may be as follows: (1) Perform multiple comparisons on each imputed dataset and therefore obtain adjusted P-values; (2) Attempt to pool (i.e., combine) the adjusted P-values generated in Step (1).

Step (1) can be done by using options offered in the LSMEANS statement in the statistical procedures (i.e., PROCs). For instance, the LSMEANS statement of PROC SURVEYPHREG offers the ADJUST= option, which you can utilize for adjusting P-values in multiple comparisons. For more about multiple comparisons, I suggest reading the book by Westfall et al. (Amazon.com: Multiple Comparisons and Multiple Tests Using SAS, Second Edition: 9781607647836: Westfa...).

Step (2) entails combining P-values in multiple imputation, a topic that has been, to the best of my knowledge, hardly discussed. However, there are some research papers on this issue. One of the commonly cited papers in a paper published in 1991 (SIGNIFICANCE LEVELS FROM REPEATED p-VALUES WITH MULTIPLY-IMPUTED DATA on JSTOR). Later on, there is a dissertation on this topic (New methods for generating significance levels from multiply-imputed data). However, I have read the entirety of this dissertation and found that it concerned combining P-values from Wald's test and other tests but did not address the issue of pooling adjusted P-values in multiple comparison procedures. So I think you can try out the method mentioned in the 1991 paper.

The last thing to note is that PROC MIANALYZE does not offer the solution of pooling P-values. It is incorrect to simply use the built-in methods (those that are called for when executing the MODELEFFECTS statement) in PROC MIANALYZE when the variable to be pooled is a P-value, as the built-in methods utilize Rubin's rule for pooling the results. Rubin's rule, on the other hand, requires that the variables to be pooled follow asymptotic normality, which is what the P-values infringe.

[GiaLee]

Hi Rob. I'm doing an analysis that adjusts p-values using the Tukey-Kramer method in a mixed model. I noticed that PROC MULTTEST doesn't support the Tukey-Kramer method. Do you know what else I could do to get a pooled adjusted p-value from PROC MIANALYZE?

[Season]

There are two issues in your problem: (1) you are trying to find out how to implement the Tukey-Kramer method; (2) you are trying to pool the P-values.

For the first issue, I searched the SAS documentation of PROC MIXED and found that its LSMEANS statement offers the ADJUST=TUKEY method. I am not an expert in multiple comparisons, so I am not that sure if that is the Tukey-Kramer method you wished to implement. I suggest you refer to the references of the SAS documentation to verify if that is the specific method you wish to implement. You can also refer to the book by Westfall et al. to find out more about multiple comparisons: Amazon.com: Multiple Comparisons and Multiple Tests Using SAS, Second Edition: 9781607647836: Westfa....

When it comes to the second issue, what can be said definitely is that PROC MIANALYZE does not support pooling P-values. Please refer to my reply to @Clemence for more details and my approaches to pooling P-values.

[GiaLee]

Thanks for your reply! Yes, I used Adjust=Tukey in the Proc Mixed model. I used Lsmeans to compare between each classess, and I can get the adjusted p-values and CI. However, I have some missing covariates that need to be handled. So, I used multiple imputation and imputed 5 datasets, and then ran the mixed model. For the p-values, I could get the pooled p-value from each comparison, but not the pooled Tukey-adjusted p-value or the adjusted CI. It seems there is no direct way to get these from Proc Mianalyze.
(I found this link very useful: https://communities.sas.com/t5/Statistical-Procedures/combine-the-LSMEANS-and-differences-in-LSMEANS...)

[Season]

Well, I have detailed my viewpoint on pooling P-values after performing multiple comparisons on each imputed dataset in my reply to @Clemence. You can refer to my reply there.

By the way, the issue of multiple imputation+multiple comparison is in fact a field hardly investigated. For the past several months, I personally have retrieved on the web and in several monographs on missing data and multiple comparisons, but has hardly found any literature addressing this issue and have not found any guideline on tackling their coincidence. You can conduct search for information by yourself, but at the end of the day, you might have to accept the fact that there is no ‘gold standard’ for the time being, at least to the best of my knowledge.

[Season]

My reply to @Clemence is on this page. You can simply scroll up to see it. In case you did not find it, I have copied it and am pasting it right here:

You mentioned a very important limitation of PROC MIANALYZE that seems to have been ignored by the majority of literature introducing this module. To the best of my knowledge, currently existent multiple comparison procedures adjust P-values on a single dataset. I have personally searched for guidelines on how multiple comparisons should be done when it is complicated by multiple imputation, which generates multiple imputed datasets. Unfortunately, I have not retrieved any guideline on that. Still, a heuristic approach to this issue may be as follows: (1) Perform multiple comparisons on each imputed dataset and therefore obtain adjusted P-values; (2) Attempt to pool (i.e., combine) the adjusted P-values generated in Step (1). Step (1) can be done by using options offered in the LSMEANS statement in the statistical procedures (i.e., PROCs). For instance, the LSMEANS statement of PROC SURVEYPHREG offers the ADJUST= option, which you can utilize for adjusting P-values in multiple comparisons. For more about multiple comparisons, I suggest reading the book by Westfall et al. (Amazon.com: Multiple Comparisons and Multiple Tests Using SAS, Second Edition: 9781607647836: Westfa...).

Step (2) entails combining P-values in multiple imputation, a topic that has been, to the best of my knowledge, hardly discussed. However, there are some research papers on this issue. One of the commonly cited papers in a paper published in 1991 (SIGNIFICANCE LEVELS FROM REPEATED p-VALUES WITH MULTIPLY-IMPUTED DATA on JSTOR). Later on, there is a dissertation on this topic (New methods for generating significance levels from multiply-imputed data). However, I have read the entirety of this dissertation and found that it concerned combining P-values from Wald's test and other tests but did not address the issue of pooling adjusted P-values in multiple comparison procedures. So I think you can try out the method mentioned in the 1991 paper. The last thing to note is that PROC MIANALYZE does not offer the solution of pooling P-values. It is incorrect to simply use the built-in methods (those that are called for when executing the MODELEFFECTS statement) in PROC MIANALYZE when the variable to be pooled is a P-value, as the built-in methods utilize Rubin's rule for pooling the results. Rubin's rule, on the other hand, requires that the variables to be pooled follow asymptotic normality, which is what the P-values infringe.

[GiaLee]

Thank you so much for the detailed information and the explanation of the problem using PROC MIANALYZE pooled p-value. I was able to get the adjusted p-value for each imputed dataset (as step 1). I will read and try the "1991" paper you mentioned. Thanks!