topic Re: effect size and confidence intervals in proc mixed in Statistical Procedures

effect size and confidence intervals in proc mixed

emaneman — Wed, 20 Jul 2022 10:25:50 GMT

Dear all,

I am using PROC MIXED to analyze data from a cluster-randomized control trial, in which I am chiefly interested in the interaction effect between a manipulation (CONDITION - a between-subject factor) and TIME (within-subject factor) - I am also adding TYPE, SEXY AND AGE as fixed factors, and I include two nested random effects: CLASS(SCHOOL) and SCHOOL(LOCATION); and one additional random effect: LOCATION. The syntax is included.

A reviewer of the paper is asking me to provide effect size and confidence interval for the critical interaction effect (CONDITION X TIME), as well as for other main effects. I have searched how to obtain this in PROC MIXED, but with little success. I have found the article by Selya et al. (2012, Frontiers), but it is not of much help in this case.

Any idea?

proc mixed ;
class id cond school class location sex time type;
model casso= 	condition|type|time|sex|ageyears   ;
repeated / subject=id type=cs;
Random intercept  /subject=class(school);
Random intercept  /subject=school(location);
Random intercept  /subject=location;

Re: effect size and confidence intervals in proc mixed

SteveDenham — Thu, 28 Jul 2022 12:56:03 GMT

One of the difficulties in finding effect size calculations in mixed models is that there is very little agreement on how to calculate them, and if they are meaningful at all when there are multiple variance components (i.e. it is difficult to calculate a pooled standard deviation to use, when that may depend on which class, school and location are utilized.).

Confidence bounds on parameters are easy enough, just add CL to the MODEL statement. Confidence bounds on the difference between conditional means is a bit harder. You'll need to use an LSMESTIMATE statement with a CL option, and correctly specify the means you are interested in. That would be easier in GLIMMIX where there is a SLICEDIFF option for the LSMEANS statement, thus enabling to compare between levels of a factor of interest at each level of some other factor or combination of factors.

SteveDenham

Re: effect size and confidence intervals in proc mixed

emaneman — Sat, 30 Jul 2022 15:17:44 GMT

Dear Steve,

thank you very much for your response to my query. The CL option in the model indeed provides what i need for the specific interaction effect of interest (conditionX time), but I wonder why i get different results for the interaction, depending on whether i specify or not the CL option. I attach the screen shot of the results I obtain with the CL option as well as without. I am obviously missing something silly.

Can you elucidate what I am missing?

Thank you in advance.

Emanuele

Re: effect size and confidence intervals in proc mixed

PaigeMiller — Sat, 30 Jul 2022 15:57:14 GMT

@emaneman wrote:

I wonder why i get different results for the interaction, depending on whether i specify or not the CL option. I attach the screen shot of the results I obtain with the CL option as well as without. I am obviously missing something silly.

The table entitled "Solution for Fixed Effects" and the table entitled "Type 3 Tests of Fixed Effects" test different things — the "Solution for Fixed Effects" produces a t-test that the fixed effect is zero, while the "Type 3 Tests" are F-tests that tests if all levels of the interaction are equal to one another. These are not the same test, and don't have to match (although in the case where an interaction has only 1 df I believe the Pr>F will match and in this case it does).

Re: effect size and confidence intervals in proc mixed

emaneman — Sat, 30 Jul 2022 15:58:25 GMT

Thank you @PaigeMiller
I am back to the issue that the reviewer wants me to provide Confidence Interval for theType 3 Tests of Fixed Effects... Is there a way to estimate the same model with PROG GLM, instead of PROC MIXED as I am currently doing (with the random factors as well)? i have tried, but do not seem to make it work.

Re: effect size and confidence intervals in proc mixed

PaigeMiller — Sat, 30 Jul 2022 16:28:09 GMT

the reviewer wants me to provide Confidence Interval for theType 3 Tests of Fixed Effects

The Type III tests of Fixed Effects are an F-test which follow an F-distribution, so I suppose it is theoretically possible to produce confidence intervals of the F-test value using the F-distribution, but I don't think PROC MIXED does that (but you could program it yourself in a SAS data step if you really want it). Furthermore, I think either you or the reviewer is confused, the confidence intervals you do want are on the Fixed effects, not on the Type 3 tests. In all my years of reading about and performing this type of modeling via PROC GLM or PROC MIXED, I cannot ever remember seeing in published papers (nor in my own work) the confidence interval on the F-test value.

I believe the meaningful thing is the confidence interval on the Fixed Effect estimates (which you have in the table). By the way the Fixed Effect parameter estimates are not Type III estimates, they are simply parameter estimates as the term Type III does not apply to parameter estimates. The Type III refers to the sum-of-squares and not to the parameter estimates.

Is there a way to estimate the same model with PROG GLM, instead of PROC MIXED as I am currently doing (with the random factors as well)? i have tried, but do not seem to make it work.

In some cases PROC GLM and PROC MIXED should provide the same results; in other cases they will not provide the same results. The question however is meaningless, as you can't get Type III estimates out of either, the "Type III" does not apply to parameter estimates.

Re: effect size and confidence intervals in proc mixed

emaneman — Sun, 31 Jul 2022 10:13:29 GMT

Hello and thank you again for your further comments!

Emanuele