Why does changing covariance structure radically change the standard error of the fixed effects?

[plf515]

Using SAS 9.4.

16 animals were randomly assigned to two treatments (control and treatment). Each animal had its level on the dependent variable measured 24 times over the course of the day. For each animal, these numbers vary wildly, from 0 into the thousands, with no strong correlations.

Interest is in whether the two treatments are different.

I compared three models:

a) T-test on total activity level

proc ttest data = wide;
 class group;
 var total;
run;

b) A multilevel model with unstructured covariances

title "MLM for day 1, unstructured covariance, only group entered";
proc mixed data = long;
 class group mouse_id;
 model act = group/solution;
 repeated/type = un subject = mouse_id;
run;

c) A multilevel model with AR(1) covariances

title "MLM for day 1, AR1 covariance, only group entered";
proc mixed data = long;
 class group mouse_id;
 model act = group/solution;
 repeated/type = ar(1) subject = mouse_id;
run;

c) Had slightly better AIC and BIC than b).

The effect size of group was roughly equal across all three models (after accounting for the fact that the t-test was on 24 values added up) but the standard errors for b) and c) were wildly different. b) Was significant, a) and c) were not.

In the past, I've not seen such huge changes in the standard errors.

Any thoughts on why this might occur and what to do?

NOTE: It's the same data, just one in long format and one in wide format.

[PaigeMiller · Accepted solution]

There's not much we can say since we don't have your data, and you have changed the data sets from one analysis to another anyway (not even sure how you can compare different data sets in this way).

But ... if you fit a model with the wrong covariance structure (which doesn't match the data), I can understand why the model doesn't fit well and the standard errors are large.