PROC GENMOD: Discrepancy in GEE and type III P-values

[abl37]

I have been told that reporting the Type III p-values is preferred over the GEE p-values as the type III p-values are more conservative. However, in my situation, the type III p-value is smaller than the GEE p-value. I have 2 questions in regards to this issue:

1) I am using manual backwards stepwise elimination to remove variables if P>0.05. In this example, My GEE p-value for Parity is 0.0904 yet the Type III p = 0.0407. Which p-value should I rely on to remove variables from my model?

2) I am reporting the risk ratio and confidence interval found in the lsmeans. If I am using/reporting the type III p-value, this p-value is not reflected in the RR and confidence interval. Is there a way of estimating the lsmeans and RR using the type III p-value? You can see in this example, the RR for parity is 1.9234 and the confidence interval is 0.9021 to 4.1010, which crosses 1 and therefore would not reflect a P-value <0.05. If I reported the type III p-value with this RR and CI, reviewers would see this discrepancy and realize that the reported p-value can't be accurate.

proc genmod data=ket;
class farm parity (ref="0") bhba (ref="0");
model ket=bhba parity/ link=log dist=poisson pscale type3;
repeated subject=farm/type=exch;
lsmeans bhba parity/pdiff cl exp;
run;

This dataset containes 1398 observations, and there are 71 clusters. I am using v. 9.4. Ket, bhba, and parity are all dichotomous variables however, ket is a rare event (3.72%). 

Any help is appreciated. Thank you. 

[MichaelL_SAS]

First I want to note that if you are using the same data to both select the model by backwards selection and then fit the model with the selected predictors, that introduces a post-selection inference issue that violates the assumptions underpinning the inference that PROC GENMOD performs for the selected model. Appropriate methods for performing post-selection inference is an active area of statistical research. Data splitting is always a valid approach if there are enough data.

To your original question, for GEE models by default the Type III tests are conducted by using the generalized score statistic. As described in the “Generalized Score Statistics” section of the PROC GENMOD documentation, this test statistic involves solving a restricted GEE model (restricted to match the null hypothesis of the test) and evaluating the generalized estimating equations at the restricted solution. This test statistic is typically preferred over a Wald based test statistic. You can request that PROC GENMOD conduct the Type III tests with the Wald test statistic by using the WALD option. Using the Wald statistic might be preferable when the data have a very large number of clusters. In that situation computing the generalized score statistics can be computationally intensive and the differences between it and the Wald test statistic should be minimal.

The LSMEANS statement computes predicted population margins, which is a different task then then testing the Type III contrasts. For the predicted population margins computation there is not an analogue to the generalized score statistic used to test the Type III contrasts.

[abl37]

Thank you for your reply, Michael.

As a follow up, it does not seem that the Type III Wald test statistic
produces a different P-value from the P-values reported in the GEE
parameter estimates table. Is the Type III wald test statistic calculated
differently? Also, in the example I showed, do you know why the wald test
statistic would be producing a more conservative (higher) p-value than the
score statistic?

[MichaelL_SAS]

The Parameter Estimates table is reporting p-values for a test where the null hypothesis is that one regression coefficient is zero. In the Type III tests, the null hypothesis is that a certain linear function of the regression coefficients is zero. Sometimes these two tests overlap, i.e. the linear function being tested is that just one coefficient is zero, in which case the Wald test statistic is calculated similarly. 

The "Four Types of Estimable Functions" chapter in the SAS/STAT documentation provides some relevant information about the Type III tests. In particular, the very last line in the "Estimability" section covers the form of the Wald test statistic and the "Type III Estimable Functions" section covers how a hypothesis for the test is constructed. I will admit, whenever I have to refer to the material about constructing the tests I have to read it (at least) two or three times before it starts to sink in. 

As to why the tests using the Wald statistics versus the generalized score statistic differ in your example, I'm not sure there's really much I can say. The number of clusters is not small but not so large to make differences between the tests surprising.