turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Find a Community

- Home
- /
- Analytics
- /
- Stat Procs
- /
- Cohen's d for interaction of factors using Porc mi...

Topic Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

11-07-2016 07:23 AM

Hi SAS community,

we are using this code for repeated measure design, including Visibility (4levels) x Congruency (2 levels) and SOA (4 levels):

PROC mixed data=d; CLASS visibility sub soa Congruency ; MODEL rt= visibility|soa|Congruency /*DDFM=sat*/ ; Repeated /subject=sub type=cs; LSMeans soa|Congruency|visibility/diff; run; quit;

I was asked to report Cohen's d for our data. I found that while for two levels factors (2 groups for comparison) one can use means and standart deviations (for proc mixed), for more complicated cases with multiple levels of factors (like 4 levels) and for interactions one needs to use square root form the MSE (mean square error) in denominator. For example, in my case for interaction between Visibility and SOA, I need to use this formula: ((Visibility1-SOA1)-(Visibility2-SOA2)- (Visibility3 -SOA3)-(Visibility4-SOA4))/(RMSE), where Visibility1, SOA2 etc. refer to means of particular levels of factors. My question is how to receive the mean square error (MSE) from the proc mixed?

Accepted Solutions

Solution

11-07-2016
10:23 AM

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to dina_d

11-07-2016 10:20 AM

Quick scary thing that I might have missed. Your equation has RMSE in it, which is a standard deviation estimate. You will have to take square roots of the residual variance component and of the sum to get things on the correct scale to calculate Cohen's d.

Complete aside--I really don't like Cohen's d for mixed models, especially when there are correlated errors. The t value for a difference tells me everything a d could, and it corrects for the correlation. It just isn't on the scale of d, and it is dependent on the number of observations.

Steve Denham

All Replies

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to dina_d

11-07-2016 09:37 AM

A first approximation is the Residual variance component. However, the standard error of the LSmeans also includes a variance component due to subject, so getting an SD to calculate Cohen's D is a bit of a problem. I would calculate two estimates of the effect size, the first using only the residual, and the second using the sum of the residual and the repeated factor.

Steve Denham

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to SteveDenham

11-07-2016 10:04 AM

Hi Steve,

thank you so much for your reply! Just a little question to make sure I understood correctly. When you talk about residual, do you refer to the residual from the "Covariance Parameter Estimates" table (of the proc mixed output)? And in our case repeated factor is subject, so does it mean that the second effect size would have in the denominator the sum of the aforementioned residual and the estimate for subject from the same table?

Again many thanks for your help!

Dina.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to dina_d

11-07-2016 10:06 AM

Hi Dina,

Yes and yes to your questions.

Steve Denham

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to SteveDenham

11-07-2016 10:11 AM

Thank you very much!

Dina.

Dina.

Solution

11-07-2016
10:23 AM

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to dina_d

11-07-2016 10:20 AM

Quick scary thing that I might have missed. Your equation has RMSE in it, which is a standard deviation estimate. You will have to take square roots of the residual variance component and of the sum to get things on the correct scale to calculate Cohen's d.

Complete aside--I really don't like Cohen's d for mixed models, especially when there are correlated errors. The t value for a difference tells me everything a d could, and it corrects for the correlation. It just isn't on the scale of d, and it is dependent on the number of observations.

Steve Denham

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to SteveDenham

11-07-2016 10:49 AM

***

Thank you for making this point about t. In fact it was hard to find measure of the size effect for mixed models. I did not think that t for the difference can be treated as an estimate of effect size. I guess there are ways to make a correction for number of observations (like d population) and to put t on d scale. Strange that there is no a special name and tutorial for calculating a better estimate of the effect size for mixed models.

Many thanks,

Dina.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to dina_d

11-08-2016 09:44 AM

Effect size as defined elsewhere for Cohen's d for continuous variables is generally the ratio of the observed difference in means to the pooled standard deviation. Once you hit mixed models, the concept of pooled standard deviation becomes murky. And, unfortunately, it can carry over to the difference in means, as can be seen by looking at various covariance structures and random effects models, especially if there is imbalance. This is because the estimates are obtained by maximizing the likelihood (or integrating the likelihood numerically) over both fixed and random effects. As a result, several of the assumptions for the standard measures of effect size are not met. Differences are conditional on the random (or repeated) effects.

This article shows how to extract Cohen's *f^2 *from PROC MIXED, which may be more appropriate here.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3328081/

Steve Denham