Computing Inter-Rater Reliability - Different subjects are scored by different researchers

[Kimberly09]

Hi Community,

I would like to test if 2 different researchers are rating different subjects in the same manner with respect to a dichotomous outcome. I know if both researchers had rated the same subject separately, I could determine the degree of agreement by the Kappa statistic. However, in this case, the 2 researchers conducted the same intervention on different subjects using the same assessment. Is there a statistical test I could use run in SAS to compare concordance between these 2 researchers? Thanks so much for your help!  

Example Data set (Actual sample size = 790)

Researcher 1                                                                 Researcher 2

subject id      completeness_rating                              subject id          completeness_rating

1001              Complete  (coded as 3)                          2001                  Complete (coded as 3)

1002              Partial  (coded as 2)                               2002                  Incomplete (coded as 1)

1003              Incomplete (coded as 1)                        2003                  Incomplete (coded as 1)

[ballardw]

Do you have pretty strong evidence that the two groups should have received a similar proportion of scores?

Chi-sq might be appropriate. But if you don't have reason to believe that the subjects distribution of scores should be similar the differences could be the subjects and not the researchers.

I would be tempted to run a quick Proc Freq requesting the chi-sq just to see.

[Kimberly09]

Thank you for your reply. No, I don't have statistical evidence as to whether the 2 groups should have similar scores. However, the groups have similar demographic and clinical characteristics and the researchers seem to think the scores should be similar. I haven't done any statistical test to determine if this is the case (so far, I've only run the usual proc freq/univariate procedures). Here comes a basic question regarding the chi-square test.....Would I use the code below to determine if there's an association between the completeness variable and site (researchers), and if so, then what would I need to do?

proc freq data=check; 
tables complete_rate*site/chisq; /*the 2 researchers are at different sites/medical facilities*/
run;