Cohen's d for interaction of factors using Porc mixed for repeated measures

[dina_d]

Hi SAS community, 

we are using this code for repeated measure design, including Visibility (4levels) x Congruency (2 levels) and SOA (4 levels):

PROC mixed data=d;
CLASS visibility sub soa Congruency ;
MODEL rt= visibility|soa|Congruency /*DDFM=sat*/ ;
Repeated /subject=sub type=cs;
LSMeans soa|Congruency|visibility/diff;
run; quit;

I was asked to report Cohen's d for our data. I found that while for two levels factors (2 groups for comparison) one can use means and standart deviations (for proc mixed), for more complicated cases with multiple levels of factors (like 4 levels) and for interactions one needs to use square root form the MSE (mean square error) in denominator. For example, in my case for interaction between Visibility and SOA, I need to use this formula: ((Visibility1-SOA1)-(Visibility2-SOA2)- (Visibility3 -SOA3)-(Visibility4-SOA4))/(RMSE), where Visibility1, SOA2 etc. refer to means of  particular levels of factors. My question is how to receive the mean square error (MSE) from the proc mixed?

[SteveDenham]

A first approximation is the Residual variance component.  However, the standard error of the LSmeans also includes a variance component due to subject, so getting an SD to calculate Cohen's D is a bit of a problem.  I would calculate two estimates of the effect size, the first using only the residual, and the second using the sum of the residual and the repeated factor.

Steve Denham

[dina_d]

Hi Steve,

thank you so much for your reply! Just a little question to make sure I understood correctly. When you talk about residual, do you refer to the residual from the "Covariance Parameter Estimates" table (of the proc mixed output)? And in our case repeated factor is subject, so does it mean that the second effect size would have in the denominator the sum of the aforementioned residual and the estimate for subject from the same table?

Again many thanks for your help!

Dina.

[SteveDenham]

Hi Dina,

Yes and yes to your questions.

Steve Denham

[dina_d]

Thank you very much!
Dina.

[SteveDenham · Accepted solution]

Quick scary thing that I might have missed.  Your equation has RMSE in it, which is a standard deviation estimate.  You will have to take square roots of the residual variance component and of the sum to get things on the correct scale to calculate Cohen's d.

Complete aside--I really don't like Cohen's d for mixed models, especially when there are correlated errors.  The t value for a difference tells me everything a d could, and it corrects for the correlation.  It just isn't on the scale of d, and it is dependent on the number of observations.

Steve Denham

[dina_d]

***

Thank you for making this point about t. In fact it was hard to find measure of the size effect for mixed models. I did not think that t for the difference can be treated as an estimate of effect size. I guess there are ways to make a correction for number of observations (like d population) and to put t on  d scale. Strange that there is no a special name and tutorial for calculating a better estimate of the effect size for mixed models.

Many thanks,

Dina.

[SteveDenham]

Effect size as defined elsewhere for Cohen's d  for continuous variables is generally the ratio of the observed difference in means to the pooled standard deviation.  Once you hit mixed models, the concept of pooled standard deviation becomes murky.  And, unfortunately, it can carry over to the difference in means, as can be seen by looking at various covariance structures and random effects models, especially if there is imbalance.  This is because the estimates are obtained by maximizing the likelihood (or integrating the likelihood numerically) over both fixed and random effects.  As a result, several of the assumptions for the standard measures of effect size are not met.  Differences are conditional on the random (or repeated) effects.

This article shows how to extract Cohen's f^2 from PROC MIXED, which may be more appropriate here.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3328081/

Steve Denham