Selection of the covariance type will depend strongly on the design. See what happens first with step 1 of your list, and see if the hetereogeneity is solved. You could then explore other covariance types. Note however that the heterogeneity in the fixed effects is a separate issue from the design, and that combining a covariance type= with group= will greatly increase the number of covariance parameters to be estimated. For instance, suppose you fit a heterogeneous autoregressive covariance structure to four points. You would estimate 5 parameters. Now suppose you add group= for a categorical variable with 6 levels. You would now have to estimate 30 parameters, 5 for each level of the categorical variable. This may lead to inability to converge, or poor estimates and standard errors, or a non-positive definite Hessian matrix. Steve Denham
... View more