How to calculate power (or sample size) for a multiple comparison when Bonferroni-adjusted p-value?

[Dennisky]

Dear all,

We conduct a study for comparing the difference of a key indicator (continuous variable) between six groups.

When continuous variables follow a normal distribution, one should use ANOVA, and when they do not follow a normal distribution, the Kruskal-Wallis non-parametric test is employed. We have completed the analysis.

Nonetheless, how can we calculate power (or sample size) for the multiple comparison experiment (ANOVA or Kruskal-Wallis non-parametric test, respectively) when using the Bonferroni-adjusted p-value method?

Thanks!

[ballardw]

Just for confirmation: Your request states "We have completed the analysis." So I take that to mean you have collected the data, done analysis and are requesting how to perform POST HOC power analysis, correct?

Anything related to "multiple comparison" you should provide details of what sort of "multiple" was involved as it would likely affect any attempt at a response.

Calculating "sample size" is sort of waste of time if you have already collected and analyzed the data.

[SteveDenham]

And to add to @ballardw 's comments, calculation of power at this point has some issues as well. Unless you are going to use this calculation to power and set a sample size for an upcoming study, I would recommend against calculating power (1 minus the probability of a type II error). Why? Because you have already collected and analyzed the data. Either you rejected the null hypothesis or you didn't. Working through the probabilities, based on this piece of information, your power is either one or zero for this dataset and analysis. I refer you to Zhang et al. (2019) http://dx.doi.org/10.1136/gpsych-2019-100069 , who finish their paper with this statement: "In general, post hoc power analyses do not provide sensible results."

SteveDenham

[Dennisky]

@SteveDenham Thank you so much.  As usual, you always bring me enlightening inspiration each time.

You and ballardw mentioned the most crucial part of this question.

However,  due to work requirements, I need try to answer this question and attempt to make some calculations. I wonder if there might be any feasible solution for it?

Thanks a lot!

[Dennisky]

Thank you. Yes, we have completed the analysis. Some reviewers asked us whether the power was sufficient. Thus, we try to calculate the power. As you said, what kind of details of what sort of "multiple" was involved do we need to provide in order to calculate this result? How can it be specifically implemented? Thanks!

[ballardw · Accepted solution]

Power calculations rely on the probability of Type 2 errors. So are dependent on the actual statistical test to be performed (if determining Power before sample collection) as the Type 2 error varies for different tests. If you perform "multiple comparisons" which can mean a lot of different things HOW those comparisons are done also can affect the Type 2 errors.

In a planning use of Power you pretty much set your sample size and calculate power for that, or set a desired power level and determine a sample size to meet that requirement, given assumptions about the distribution of the data and significance levels of given tests.  Note that power will depend on sample size or vice versa. So "calculating" power and sample is sort of a misnomer. Prior to a study you do what-if sort of things like "how does the sample size change with power" or "how does power change with sample size" . BUT those calculations still depend on the specific test(s) to be performed and the distribution assumptions.

---

@Dennisky wrote:

Thank you. Yes, we have completed the analysis. Some reviewers asked us whether the power was sufficient. Thus, we try to calculate the power. As you said, what kind of details of what sort of "multiple" was involved do we need to provide in order to calculate this result? How can it be specifically implemented? Thanks!

---