Mixed Procedure with METHOD=TYPE3

[JeremyPoling]

I have a very basic question about the MIXED procedure.  Suppose you use the METHOD=TYPE3 option in the PROC MIXED statement and the DDFM=SATTERTH option in the MODEL statement.  What is the difference between the F tests for fixed effects displayed in the ANOVA table and the tests displayed later in the "Type 3 Test for Fixed Effects" section of the output?  Are both test valid?  Is one test preferred over the other?  Thanks in advance.

[1zmm]

The METHOD=TYPE3 option in the PROC MIXED statement specifies to the procedure how to estimate variance component models (covariance parameters) using expected mean squares (methods-of-moment estimates)(compare with PROC GLM estimates).

The DDFM option in the MODEL statement specifies how the denominator degrees of freedom are calculated for estimating significance probabilities for the fixed effects.

The tests displayed in the table, "Type 3 Test for Fixed Effects", refer to tests of the fixed effects and are unrelated to the tests above of the variance components.   These fixed-effects tests can be calculated even if the METHOD option is not a TYPEn option (see above).  Type 3 tests "adjust" for all the fixed effects in the model.  Because these two kinds of tests refer to two different parts of the model, both provide different kinds of information and are not preferred to one another.  Method-of-moment estimates of variance components are less preferred nowadays than restricted maximum likelihood or maximum likelihood estimates.(METHOD=REML or METHOD=ML).

[JeremyPoling]

Hi.  Thanks for your response.  However, I still don't completely understand.

The main reason for using the METHOD=TYPE3 option instead of REML is the ANOVA table in the output.  Some folks still want to see the sum of square breakdown.  Call them old-fashioned I guess.  The ANOVA table contains tests for both fixed and random effects.   My question relates to how the tests for fixed effects in this table relate to the "Type 3 Tests for Fixed Effects."

You said that "The tests displayed in the table, "Type 3 Test for Fixed Effects", refer to tests of the fixed effects and are unrelated to the tests above of the variance components."  Maybe I am misunderstanding something, but I don't believe that this statement is accurate.  I'm pretty sure that the estimated V covariance matrix, based on the method-of-moments estimators, are used to construct the test statistic.  Therefore you would not get the same results of these tests if you used a different method of estimating the covariance parameters.  I specified the DDFM=SATTERTH option because I believe that PROC GLM (or PROC MIXED with the METHOD=TYPE3 option) uses the Satterthwaite approximation whenever the error term for one factor in the model does not equal the expected mean square value of another factor.  I was hoping that this option would make the two test equivalent, but no dice.

From what I can tell, the test in the ANOVA table and the "Type 3 Test for Fixed Effects" are always the same when the design is balanced, but I'm not positive.  I know that the tests in the ANOVA table are not exact when you have unbalanced data.  Does anyone know if the other test is an exact test?  Also, I can recreate the "Type 3 Test for Fixed Effects" using a CONTRAST statement, but not the ANOVA test.  I tried to look through the SAS documentation on the fixed effects test, but I only found a mess of complicated matrix equations.  It's been about seven years since I've last opened my linear models textbook, and I'm hoping that I won't need to now as I'm more than a bit rusty.

I have a situation where the p-value is slightly less than the significance level on one test and slightly more than the significance level on the other test.  If possible, I would like to justify using one test or the other.

Thanks again,

Jeremy

[SteveDenham]

I was understanding your frustration right up to:

I have a situation where the p-value is slightly less than the significance level on one test and slightly more than the significance level on the other test.  If possible, I would like to justify using one test or the other.

Is there something I am missing here?  Not wanting to go fully Bayesian on you, but a p value is only a probability, given that the null hypothesis is true.  Since it is nigh on to impossible that Ho is actually true, the p value is really only an approximation (even for so-called exact tests).  If one test is less than 0.05 and another is greater than 0.05, picking a test after running the test really messes up decision making.  Why not just report the p value, and make any statements regarding inference based on that?

I tend to agree with 1zmm regarding method of moments estimators and tests, probably because it has been shown that the REML estimates are more consistent.  I sympathize that someone wants to see sums of squares, but that doesn't necessarily mean that they are appropriate.  What is the design in this case?

Steve Denham

[lvm]

I really think you should stick with the REML estimation and testing, and not use method=type3 (based on expected mean squares). Contemporary mixed modeling has really moved away from sum of squares and mean squares. Although the latter can be fine for the simplest of situations, the expected mean square approach cannot be generalized for any of the interesting problems that are common. The likelihood-based analysis is the way to go. When I give workshops on mixed models, I tell the audience to forget everything about mean squares.

[JeremyPoling]

Steve,

I probably could have chosen my words better.  I agree that it's not appropriate to pick a test after running the test, so there's no need to open up a can of Bayesian stats on me right now  

With the METHOD=TYPE3 option, SAS produces two different tests of the same hypothesis that can have different results when the design is unbalanced.  I just want to understand, on a high level, what the difference is between the two tests.  If one test is not valid or one test is preferable to the other, then I want to know that.

It is a very basic two-way crossed unbalanced design with one fixed factor and one random factor.  Here is a silly example that is somewhat analogous:

proc mixed data=sashelp.shoes method=type3;

     where region in ('United States" "Western Europe");

     class region product;

     model sales = region/ddfm=satterth;

     random product region*product;

quit;

Thanks again;

[lvm]

If you want a detailed explanation, you need to read some textbooks, or parts of textbook. I doubt if postings will go into the details. The book SAS FOR MIXED MODELS, 2nd edition (2006) is a great reference. Here are a few things to consider. For PROC MIXED, the default is a likelihood approach (REML). This should tell you something. Here is another way to look at the problem. Method=type3 essentially duplicates the approach in PROC GLM, and the book I mentioned above makes it clear why one should not use GLM for models with random effects. The GLM approach (i.e., the method=type3 approach) actually fits the model with all terms as fixed effects, whether or not they are really random effects. Then, in a post-model-fitting step, expected mean squares are calculated and tests conducted based on the theoretical values of the mean squares. This approach (everything fixed to start with, then adjust later) can give appropriate results for a narrow set of circumstances (such as for balanced data, all effects independent, and so on); for this narrow set of circumstances, the method=type3 approach will duplicate the method=reml approach. Otherwise, the results will be different. For REML model fitting, the random effects are treated as random effects in the model-fitting process (as one would want, since they are random effects), and the tests of significance are derived based on contrasts of the expected values (there are no mean squares).

The biggest effect of the model-fitting choice will show up in the estimated standard errors for estimated means and contrasts of means. For certain designs (e.g., split plots) , whether balanced or not, and for most (all?) unbalanced layouts, the standard errors will be incorrect when using method=type3, but correct with REML.

So, I personally recommend likelihood-based methods for mixed models. But as I said above, there is a narrow set of circumstances where the results will be the same. The book I mentioned will also show you some situations (within this narrow set of circumstances), where valuable information can be obtained by using the type3 approach.

[SteveDenham]

Adding on to what said--if you have random effects, and you want to truly consider them as random (i.e., they are a sample of all possible realizations of the random variable), and you need to infer to that broad inference space, then you never want to use TYPE3.  Never.  Under method=type3, the results are only applicable to repeated realizations of your data on the specific levels of the random variable for which you already have observations.

So, in this case, one test is definitely preferred over the other.  In addition, should you wish to specify any structure to the variance-covariance matrix, you can't use method=type3.  If you wish to specify subject= in the random statement, you can't use method=type3.  If you wish to use a repeated statement, you can't use method=type3 (at least in the one try I threw at it).

Steve Denham

[JeremyPoling]

Larry and Steve,

First of all, thanks so much for your responses.  I sincerely appreciate you both for taking the time to think about this problem and respond. A coworker of mine has the book that Larry referenced.  I'll take a look at it.

While PROC GLM does always fit a fixed effect model, I don't believe that the same is true for PROC MIXED with the METHOD=TYPE3 option.  Here are several observations that make me believe that this is the case:

1. The default ANOVA table created with PROC GLM is completely incorrect.  You need to specify the TEST option on the RANDOM statement to reproduce the tests created by PROC MIXED with the METHOD=TYPE3 option.
2. With the G option on the RANDOM statement, PROC MIXED with METHOD=TYPE3 will print the estimated G covariance matrix.  If PROC MIXED fit a fixed effects model, then the G matrix does not exist.
3. With the SOLUTION option on the RANDOM statement, PROC MIXED with METHOD=TYPE3 will produce EBLUPs for the observed levels of the random effects.  I don't think that it's possible to create EBLUPs with a fixed effects model.
4. ESTIMATE and CONTRAST statements involving only fixed effects produce totally different results in PROC MIXED with METHOD=TYPE3 than in PROC GLM.  This is because PROC GLM does not correctly compute the standard errors.

Also, I think that PROC MIXED with METHOD=TYPE3 does use the broad inference space while PROC GLM uses only the narrow inference space.  To see all this, compare the output produced by the following two steps.  Note that the "U.S. - Narrow Space" estimate in PROC MIXED with METHOD=TYPE3 is equivalent to the ESTIMATE statement produced by PROC GLM.  (Sorry if there are any typos in the code, I don't have SAS on this computer.)

proc mixed data=sashelp.shoes method=type3;

    where region in ('United States' 'Western Europe');

    class region product;

    model sales=region/ddfm=satterth;

    random product region*product / solution g;

    estimate "U.S. - Broad Space" intercept 1 region 1 0;

    estimate "U.S. - Narrow Space" intercept 8

                                                   region 8 0 |

                                                   product 1 1 1 1 1 1 1 1

                                                   region*product 1 1 1 1 1 1 1 1 /divisor=8;

quit;

proc glm data=sashelp.shoes;

     where region in ('United States' 'Western Europe');

     class region product;

     model sales = region product region*product;

     random product region*produce/test;

     estimate "U.S. with GLM" intercept 1 region 1 0;

quit;

Based on these results, I'm inclined to believe that the tests created with PROC MIXED and METHOD=TYPE3 are for valid mixed effect models.  Am I still missing something here?

I don't argue with any of the virtues of using REML over the EMS method.  I used METHOD=TYPE3 in this particular application only because the report I was generating has an expected format that includes an ANOVA table.  Whether or not this format is appropriate is certainly debatable, but that was not my call.  In fact, my question has nothing to do with REML at all.  I'm not trying to compare one test generated with REML estimates of the VCs to another test that uses EMS estimates of the VCs.  I'm only using the EMS approach via the METHOD=TYPE3 option.  My question is about the two tests of the same hypothesis that are both generated with the METHOD=TYPE3 option.

Thanks again. 

[lvm]

Glad you are studying the problem in such detail. Notice that I started my previous post with "If you want a detailed explanation, you need to read some textbooks, or parts of textbook. I doubt if postings will go into the details." You misunderstood some of my comments, maybe because I couldn't write enough. Probably I was too casual in a few places.  I can only comment on a few of your statements here. GLM does a purely fixed-effects analysis, with post-model-fitting adjustments to get the random effects variances. Of course, you must use the right code (test option on random statement) to get the proper mean squares and F tests (the defaults clearly are wrong, as you point out). GLM is not capable of getting the right standard errors in many cases (with any options), especially when the SEs involve combinations of variances. This is why the estimate and contrast statements are totally wrong in GLM. GLM should be avoided when there are random effects. This was one of the points I was trying to make.

When you use method=type3 in MIXED, you are getting, in part, the equivalence of the approach in GLM (assuming you used the right options in GLM), but it is not necessarily done in the same way. The expected mean square table, F statistics (based on the mean squares), and P values agree for the two procedures (once again, assuming you have the right code for GLM and method=type3 in MIXED). This is true for your example (by the way, you misspelled product in the interaction term in GLM). This is a reason why I said you were getting a fixed-type analysis, but clearly it is not an actual fixed effect analysis in MIXED. MIXED is much more sophisticated than GLM; MIXED is doing an actual mixed-effect analysis for all the method= options we have been discussing (when you have random statements). Sorry if I implied otherwise. Because MIXED knows how to use the random effects, you obviously can get a G-side covariance matrix. You are getting appropriate SEs, ESTIMATEs, and CONTRASTs with MIXED, but not necessarily the best choices with the method=type3 analysis (the latter may be open to some interpretation). Because MIXED is properly using random-effects terms as random effects, you get EBLUPs, and so on. Obviously, these do not exist when using GLM (one of many reasons not to use GLM). There is no point even trying to recover anything like an EBLUP in GLM.

MIXED is a great procedure, and one you should be using (whether or not you use expected mean squares). The book I mentioned (SAS for Mixed Models, second edition) has several examples where method=type3 is utilized for interesting purposes. The limitation is that the method is appropriate for only a narrow class of mixed models. I would argue that likelihood-based methods are better for other reasons, but others could disagree.

I did not bring up anything about narrow or broad inference. I think you can do either with the type3 method, with the right syntax. I don’t know if anyone has evaluated the effects of the model-fitting method on the accuracy of the results for broad and narrow inference.

[lvm · Accepted solution]

Additional follow-up: With all the commentary, I am not sure if you got your original question answered. MIXED does fixed-effect hypothesis testing based on Wald statistics. For instance, if treatment has 6 levels, 5 contrasts are generated (in the background) and are simultaneously tested with an F test to determine if treatment has an effect. Different options control the df. However, if you specify method=type3, in addition to the tests based on Wald statistics, you get a second table of test results (as you noticed). These are based on the traditional approach taught in classical ANOVA textbooks. Ratios of mean squares are calculated based on the theoretical expected mean squares, and these ratios would equal 1 if the null hypotheses were true. I don't think one can easily say which test results are most appropriate. Under a narrow set of conditions, the tests based on ratios of mean squares are exact (in one sense, at least). But in more general situations, it is a tougher call. I am sure that others have compared type I and II error rates for these two approaches, but I can't think of any right now. Someone else may know. Since my own approach to mixed models is likelihood based, I tend to favor the usual Wald statistics.

[JeremyPoling]

Thanks again.  This is exactly the answer that I was looking for.