I am also currently studying multiple comparison procedures. I did not know that PROC MULTTEST can plug-in raw P-values before. That is a very useful piece of information. @SAS_Rob wrote: You would have to use a different adjustment method than Tukey like Bonferroni since MULTTEST will not compute Tukey p-values from a data set of raw p-values. Bonferroni methods are conservative. Are there any macros capable of plugging in raw P-values to implement Tukey's method? Thanks! ... View more

There are two issues in your problem: (1) you are trying to find out how to implement the Tukey-Kramer method; (2) you are trying to pool the P-values. For the first issue, I searched the SAS documentation of PROC MIXED and found that its LSMEANS statement offers the ADJUST=TUKEY method. I am not an expert in multiple comparisons, so I am not that sure if that is the Tukey-Kramer method you wished to implement. I suggest you refer to the references of the SAS documentation to verify if that is the specific method you wish to implement. You can also refer to the book by Westfall et al. to find out more about multiple comparisons: Amazon.com: Multiple Comparisons and Multiple Tests Using SAS, Second Edition: 9781607647836: Westfall PhD, Peter H., Tobias PhD, Randall D., Wolfinger PhD, Russell D.: Books. When it comes to the second issue, what can be said definitely is that PROC MIANALYZE does not support pooling P-values. Please refer to my reply to @Clemence for more details and my approaches to pooling P-values. ... View more

You mentioned a very important limitation of PROC MIANALYZE that seems to have been ignored by the majority of literature introducing this module. To the best of my knowledge, currently existent multiple comparison procedures adjust P-values on a single dataset. I have personally searched for guidelines on how multiple comparisons should be done when it is complicated by multiple imputation, which generates multiple imputed datasets. Unfortunately, I have not retrieved any guideline on that. Still, a heuristic approach to this issue may be as follows: (1) Perform multiple comparisons on each imputed dataset and therefore obtain adjusted P-values; (2) Attempt to pool (i.e., combine) the adjusted P-values generated in Step (1). Step (1) can be done by using options offered in the LSMEANS statement in the statistical procedures (i.e., PROCs). For instance, the LSMEANS statement of PROC SURVEYPHREG offers the ADJUST= option, which you can utilize for adjusting P-values in multiple comparisons. For more about multiple comparisons, I suggest reading the book by Westfall et al. (Amazon.com: Multiple Comparisons and Multiple Tests Using SAS, Second Edition: 9781607647836: Westfall PhD, Peter H., Tobias PhD, Randall D., Wolfinger PhD, Russell D.: Books). Step (2) entails combining P-values in multiple imputation, a topic that has been, to the best of my knowledge, hardly discussed. However, there are some research papers on this issue. One of the commonly cited papers in a paper published in 1991 (SIGNIFICANCE LEVELS FROM REPEATED p-VALUES WITH MULTIPLY-IMPUTED DATA on JSTOR). Later on, there is a dissertation on this topic (New methods for generating significance levels from multiply-imputed data). However, I have read the entirety of this dissertation and found that it concerned combining P-values from Wald's test and other tests but did not address the issue of pooling adjusted P-values in multiple comparison procedures. So I think you can try out the method mentioned in the 1991 paper. The last thing to note is that PROC MIANALYZE does not offer the solution of pooling P-values. It is incorrect to simply use the built-in methods (those that are called for when executing the MODELEFFECTS statement) in PROC MIANALYZE when the variable to be pooled is a P-value, as the built-in methods utilize Rubin's rule for pooling the results. Rubin's rule, on the other hand, requires that the variables to be pooled follow asymptotic normality, which is what the P-values infringe. ... View more

I saw your screenshots and codes and found it was done on R. This is a forum of SAS users. SAS and R are distinctly different statistical softwares. It is true that many SAS users are also familiar with R, but there are also many (e.g., me) who hardly know anything about R. It would be better if you post your questions in forums of R users to get a more rapid answer. ... View more

@PaigeMiller Well, first of all, thank you for your reply. I skimmed through your replies and found it may match my requirements. It feels good that someone ultimately attempts to address the problem after so many replies. But I think it unfair to claim that someone has not searched for answers without investigating if the person has actually performed any search. Well, I would like to say that I had not raised questions on that, but I did had searched on this Community and retrieved no usable result. It is then when I posted this suggestion online. Still, I think that softwares easy to use are better. I have not yet verified if the ODS tactic you mentioned can actually help to solve the problem, but even if it can, consider the long learning curve of the vast amount of knowledge concerning ODS as well as the formidable amount of options and codes revolving it. Only a person skillful enough can handle the circumstance I mention in a rapid manner. Considering the fact that arranging the panels with designated number of columns in each row is such a common issue, wouldn't it be better if a more user-friendly alternative can be found to green-handers of SAS to that more people wish to embrace it? ... View more

First of all, thank you for your compliments. It is a pleasure discussing statistical issues with you. What is more, I happened to log in my SAS account tonight to download appendical materials for a SAS book I was reading. Surprisingly, I found that the majority of the number of times of the our discussions reached a staggering level of some 2000, outnumbering virtually almost any other appearances on this Community. It seems the topics we have been discussing is of particular interest to a great number of people, which further motivates me to go on the discussions. However, I would like to say a big "sorry" for responding to your latest questions so late. I was very, very, very busy last October, when our discussions took place. Now I have finished the projects I had been working on, so I have time to discuss the issues you mentioned in depth with you. @JanetXu wrote: Now only for the p-value. So, in order to get a pooled p-value after imputation, pooling 'z' is the correct way.  In the proc mianalysis, the estimate of my pooled 'z' is just the simple mean (Robin's rule is this, right), which should follow normal, by central limit theorem, correct? then my question will be:  'what quantity' follows the non-central t?   It seem there is a contradict.    I am afraid that you had not been quite familiar with imputation by the time you posted your replies. First of all, the paragraph consisted of several small mistakes. It is "Rubin's rule" instead of "Robin's rule" that was applied. This rule was named after Donald Rubin, who made remarkable contributions to the field of multiple imputation. In addition, the module handling the pooling process in SAS is called "PROC MIANALYZE". Second, let me explain the pooling process further. Rubin's rule refers to pooling the variables of interest after calculating their values separately on each and every of the imputed datasets. Let m denote the number of imputed datasets (i.e., the number of times you impute), Q denote the variable you are interested in knowing its true value in the population but is complicated by missing values and Q1, Q2, ..., Qm denote the values of that varaible you calculate from each imputed dataset. To apply Rubin's rules, Q1, Q2, ..., Qm should follow t distributions (please note that it is not non-central t distributions) or have asymptotic normality. If that condition holds (i.e. is satisified), then you estimate Q by U, which equals (Q1+Q2+...+Qm)/m. U, on the other hand, follows a t distribution. Hypotheses testing on Q can be substituted by those on U, generating P-values, which is something that you want in the first place. Now that U  follows a t distribution, the P-values are calculated by referencing the value of U against a t distribution. Let us return to your specific question now. The central limit theorem concerns the distribution of means, which has little to do with U. You can of course argue that U is in fact a particular type of mean, so it should follow the central limit theorem as well. That is true, but in both theory and practice, we should reference U against t instead of normal distributions. By the time you read this line, you can skim through the passages I have wrote and try to find out a place where non-central t distributions appears. There is nowhere, right? So in the entire framework that a practitioner should master for handling missing data, no variable follows a non-central t distribution. @JanetXu wrote: Maybe, the answer is, following normal is approximate, following t is precise ? No, not true. When we apply Rubin's rules, we always reference U against t distributions. @JanetXu wrote: And for p-value, what is the p-value from SAS output for,  for the 'some qunatity' follows the non-central t, correct?   Since my estimate for my pooled z follows normal, why should I report that p-value? The answer to the first question is "not true" again. Please refer to my elaboration on Rubin's rules for details. As for the second question, the pooled z's (which I denoted as U) follow a t distribution. ... View more

Thank you, Steve, for your valuable information! Yes, it is the residuals rathers than the dependent variables that should follow (multivariate) normal distribution(s). I was taught in class to examine the dependent variables themselves, which is incorrect. ... View more

@Season wrote: You can safely conclude that all of the pooled parameters in PROC MIANALYZE follow a t distribution, regardless of the distribution of the original parameters to be pooled. Erratum: A mistake was made here. The correct sentence is: "You can safely conclude that all of the pooled parameters in PROC MIANALYZE follow a t distribution when univariate instead of multivariate hypothesis testing is to be done, regardless of the distribution of the original parameters to be pooled". Sorry for the mistake. ... View more

@PaigeMiller Thanks for reminding! The paper you suggested yesterday will suffice. Algorithm link: PLS generalised linear regression    ... View more

@ballardw Thanks for your reply. I admit that my judgement concerning whether the ugliness of left justification is my own opinion. But I have cited circumstances where my concern applies to. If I am presenting five calibration plots to an editor of medical profession, his/her top concern may not be the justification of the axes. Instead, he/she may demand me that I put my graphs in a nicer position. In this case, I must follow his/her order, for he/she is the editor and I am not. I don't think such circumstances are rare. In addition, the user can specify options in PROC SGPANEL to make sure that the axes are of the same starting and ending values. So specifying the justification does not equal to misinterpretation of results, which is at least part of the concern of justifying the axes. Thanks for sharing the link regarding GTL. But don't you think the formidably lengthy codes are inferior to simply adding options in the original PROC? Why bother learning and typing that much, since what I request is not complicated at all? The users should be granted more freedom to choose the format of displaying their graphs in an easy manner. This is my concern. I think my concern is neither complicated nor advanced to that extent that GTL has to be applied to solve the problem, which is, unfortunately, the case right now. It's OK if you think left-justifying all the graphs is appropriate, but the users should be given a second choice in an easy way. Regarding the SAS GRAPH modules, thank you for reminding me of their presence. I had simply forgotten all about them when I was plotting them, for when I compared a scatterplot drawn with both PROC GPLOT and PROC SGPLOT, the "ancient" feel of the plots generated by PROC GPLOT halted my further exploration into this ancient module and any module of this sort. ... View more

@JanetXu wrote: 2) "that the distribution of the parameters to be pooled differs from that of the pooled parameter." "You can safely conclude that all of the pooled parameters in PROC MIANALYZE follow a t distribution, regardless of the distribution of the original parameters to be pooled."  For my case, i.e., I am pooing my 'z' and my pooled 'z' follows a non-central t.  No, the last sentence is incorrect. Your z-statistics follow a standard normal distribution, which is not the same as a non-central t distribution. @JanetXu wrote: So, I am still having question. The p-value from proc mianalyze for my pooled 'z' is what I should report? Yes, that's correct. @JanetXu wrote: What I meant is: we have missing data; we did MI; we got m imputed data sets; we got m 'z' from wilcoxon test; we got a pooled mean from proc mianalyze, which is just simple mean of all of my 'z' (but the pooled mean follows a non-central t); I got a p-value from proc mianalysis, which is for how the t is away from the non-central t (that is H0).  Then, in the end, I should report this p-value (from proc mianalyze) as the 'best guess' p-value for our original data?   Your questions centers two topics: (1) Point estimate of the central tendency of the populations; (2) Hypothesis testing of populations. Regarding the first question, I don't think reporting the combined means is a good choice. Please check the normality of your epilepsy scores (via the complete cases). It is often because of violation of normality that the data analyst resort to Wilcoxon sum-of-rank tests to test intergroup differences. In this case, reporting the means of the two groups is inappropriate. Instead, you should report the medians. That is, you should calculate the medians of each imputed dataset and pool them as the eventual estimate of the central tendency of your populations. By the way, 1) according to central limit theorem, the means of any sample never follow a distribution other than the normal distribution, so your sentence "the pooled mean follows a non-central t" is incorrect; 2) there is currently no existing guideline as to how to pool the medians of the imputed dataset. Not long ago, I asked another user in the Community about that question. Here's the link of his/her answer: Missing value imputation. You can have a look. Regarding the second question, as I had said before, you can directly report the result PROC MIANALYZE presented to you. In short, you should report the pooled medians as a measure of central tendency and the P value PROC MIANALYZE presented to you as the result of hypothesis testing of intergroup difference. That is, your P value is not calculated from something regarding the median. @JanetXu wrote: 3) Explanation about my original thinking:  I was thinking to get a ‘pooled’ statistic on my own from SAS output window, either from 'sumofscore" or "expectunderH0' (words from sas output window; maybe a little differences from sas output data set), also use Std Dev Under H0, etc . Later I thought I can directly pool 'z'.  Till this moment, I am still not sure which method is the 'best'. Pooling the z-statistic is the best among the three statistics you mentioned. As I had stated before, the sum-of-ranks and expected sum-of-rank under H0 do not follow a normal distribution, which is what violates Rubin's rules of pooling the results of each imputed dataset.   @JanetXu wrote: 4)  About stacking data directly:  "Of course you can stack the datasets, but the correct way of dealing with missing data via multiple imputation (MI) is to calculate the statistics separately in each imputed dataset and combine (pool) them in some way." I know the generay way is not 'stacking'.  But, for our data, what we imputed is the missing seizure count for missing days. Our endpoint is percent change of seizure frequency (averaged to per 28-day; there is some calculation after the imputation.).  I read the book of "flexible imputation of missing data" by Buuren; it said a few sentences about stacking data directly.  "If the scientific interest is solely restricted to the point estimate, then the stacked imputed data can be validly used to obtain a quick unbiased estimate for linear models. Be aware that routine methods for calculating test statistics, confidence intervals, or p-values will provide invalid answers if applied to the stacked imputed data."  No time to think over and no mature idea.  I just feel that, simple 'stacking' and getting 'averaged' seizure count of each day from m imputed data sets seems making some sense.   There are three concerns regarding these words. The first one is about van Buuren's words, he said that stacking the datasets and report the point estimate of the stacked dataset is a choice in linear models, but Wilcoxon sum-of-rank test is not a nonparametric method instead a linear model. Secondly, I don't know if this validity van Buuren stated concerning point estimate carries to hypothesis testing. Thirdly, you had stated that your endpoint was the percentage of change from baseline in seizure frequency, so it would be better to stick to your endpoint and conduct point estimate as well as hypothesis testing during the entire process of data analysis instead of deviating from it in the first place, try a different method and somehow try to reach your goal eventually in a detour as long as directly reaching your goal is a viable choice. ... View more

@Quentin Thanks for your reply. I don't think merely having the graphs in columns and rows with shared axes for easy comparisons is suffice. It is often required by journals that the graphs be arranged in panels when there are multiple graphs. In situations like these, the only PROC I can resort to is PROC SGPANEL. However, in my humble opinion, a graph with two panels left-justified in the second row is ugly when there are already three in the first row. It should therefore never be used for formal presentation. You mentioned PROC SGPLOT, but unfortunately PROC SGPLOT does not support putting multiple graphs in panels. I had tried it before. You mentioned ODS LAYOUT. I will try that option to see if it works.               Finally, you mentioned GTL. This relates to another reason why I posted my complaints online. I think GTL is too difficult to learn. Assigning the number of graphs and the justification in each row is not a complicated and advanced request. I think the users should be granted more flexibility and freedom to draw their own plots in an easy manner.       ... View more

All right. Thank you for your attention paid to and time spent on me! ... View more