McNemar's Test for Propensity Matched Data and Type I Error

[thanksforhelp12]

Hello, 

I am attempting to use McNemar's test to assess for differences between a treatment and control group in a retrospective propensity-matched study. Each group has about 1000 patients and the propensity matching was performed using proc psmatch.

My question is that when I use McNemar's test using the "agree" option in the PROC FREQ command, it shows that every single outcome measure has P<0.001. Things where chi-square shows P=1.0 are significant P<0.001. A difference of 10.79% verses 10.71% is P<0.001 with McNemar and P=1 with chi square. Is this normal?

I understand the premise behind the paired nature of McNemar's test but I just have never never seen such massive differences between paired and unpaired statistical tests and feel as though something is wrong. Thanks!

[thanksforhelp12]

Any thoughts?

[FreelanceReinh]

Hello @thanksforhelp12,

Sorry to see that nobody has replied to your post yet.

Let me first say that I'm not familiar with propensity matching, so I don't really know what is "normal" under these circumstances.

My thoughts as a mathematician are:

Let nij (i=0, 1; j=0, 1) denote the numbers of matched pairs with the four possible outcomes for a binary outcome measure, where i is the result for the subject in the control group and j the result for the treated patient. So, n00, n01, n10, n11 form the 2x2 contingency table that you analyzed with PROC FREQ (at least for the McNemar test) and the total n:=n00+n01+n10+n11 is about 1000 in your study, right?

The McNemar test statistic (n10-n01)²/(n10+n01) must exceed cinv(0.999, 1), approx. 10.83, in order to obtain a p-value <0.001.

An evaluation of the two groups as if they were independent samples would be based on the contingency table with entries n00+n01, n10+n11, n00+n10 and n01+n11 (having overall total 2n). The Pearson chi-square test statistic for this table can be zero (hence p-value 1) only if (n00+n01)(n01+n11)=(n10+n11)(n00+n10). An easy calculation shows that this is equivalent to n01=n10 and hence incompatible with a non-zero McNemar test statistic (see above). Indeed, I found that the Pearson chi-square can be written as 2n(n01-n10)²/(n²-(n11-n00)²) and that its smallest possible value with n=1000 and p(McNemar)<0.001 is about 0.2420 (p-value 0.6228), e.g. with n00=494, n01=0, n10=11, n11=495.

Did you perhaps calculate the Pearson chi-square for the original contingency table (n00, n01, n10, n11)? In this case it is possible to obtain a p-value (close to) 1, especially if the outcomes of the treated and control subject are close to independent.