These are great questions, thank you. Here is an example of 14 dichotomous attitudinal items with N = 684 participants with complete data. EAP estimates were used for the ability scores. Although not an ideal measure of unidimensionality, the first two eigenvalues are 12.374 and 0.937 indicating one prominent factor. I've attached my PROC IRT code and how I created the 10 decile groups or bins using SAS. The hand calculations are below, along with the Yen's Q test statistics from the PROC IRT program. Item parameter values are provided in the SAS program. I can provide more detail if helpful. If it was a difference of 94.23 vs 97.38, I would think it is just a difference in "bin assignment". But the differences are huge (e.g., item 1 - 133.42 vs 17.36). And the formula is so simple... I can't figure out what we are doing wrong, but I don't want to keep working on it once we realized that changing the estimation method for theta in PROC IRT (e.g., using EAP vs ML vs MAP) did not change the item fit statistics in PROC IRT - which it would have to if the original version of Yen's Q (and LR G-square) are being used. So we knew we had to figure this out first. P(u1 = 1 | q) p-value Yen's Q is sum Phi Ohi Nhi rhi = Ohi - Phi rhi2 Phi*(1-Phi) Nhi*rhi2 / (Phi*(1-Phi)) Item1 Item1 raw Frequency per Bin Decile 1 0.0057 0.0147 68 0.009 0.000 0.006 0.972 Decile 2 0.3463 0.2941 68 -0.052 0.003 0.226 0.819 Decile 3 0.7515 0.8116 69 0.060 0.004 0.187 1.335 Decile 4 0.8974 0.7794 68 -0.118 0.014 0.092 10.283 Decile 5 0.9713 0.8696 69 -0.102 0.010 0.028 25.601 Decile 6 0.9560 0.9706 68 0.015 0.000 0.042 0.345 Decile 7 0.9994 0.9706 68 -0.029 0.001 0.001 94.060 Decile 8 0.9999 1.0000 69 0.000 0.000 0.000 0.007 Decile 9 1.0000 1.0000 68 0.000 0.000 0.000 0.000 Decile 10 1.0000 1.0000 69 0.000 0.000 0.000 0.000 Yen's Q 133.421 P(u1 = 1 | q) Yen's Q is sum Phi Ohi Nhi rhi = Ohi - Phi rhi2 Phi*(1-Phi) Nhi*rhi2 / (Phi*(1-Phi)) Item2 Item2 raw Frequency per Bin Decile 1 0.0000 0.0000 68 0.000 0.000 0.000 0.000 Decile 2 0.0000 0.0000 68 0.000 0.000 0.000 0.000 Decile 3 0.0000 0.0000 69 0.000 0.000 0.000 0.003 Decile 4 0.0008 0.0147 68 0.014 0.000 0.001 15.989 Decile 5 0.0689 0.1739 69 0.105 0.011 0.064 11.870 Decile 6 0.7439 0.7059 68 -0.038 0.001 0.191 0.515 Decile 7 0.9982 0.9412 68 -0.057 0.003 0.002 122.961 Decile 8 1.0000 1.0000 69 0.000 0.000 0.000 0.000 Decile 9 1.0000 1.0000 68 0.000 0.000 0.000 0.000 Decile 10 1.0000 1.0000 69 0.000 0.000 0.000 0.000 Yen's Q 151.339 Although P_hi appears to be 0 or 1 for some bins, these are asymptotic to the lower and upper limits of 0 and 1, thus we made sure the values (e.g., .000021) are retained in the cells. Item Fit Statistics Item DF Pearson Pr > P ChiSq LR Pr > LR ChiSq Chi-Square Chi-Square Item1 8 17.36219 0.0266 19.54714 0.0122 Item2 8 2.6954 0.952 3.12093 0.9265 I didn't post the actual data because it falls under a MDUA data restriction. But if I need to create a new example, I will.
... View more