topic Re: Computing Inter-Rater Reliability - Different subjects are scored by different researchers in Statistical Procedures

Computing Inter-Rater Reliability - Different subjects are scored by different researchers

Kimberly09 — Thu, 04 Jun 2020 19:34:28 GMT

Hi Community,

I would like to test if 2 different researchers are rating different subjects in the same manner with respect to a dichotomous outcome. I know if both researchers had rated the same subject separately, I could determine the degree of agreement by the Kappa statistic. However, in this case, the 2 researchers conducted the same intervention on different subjects using the same assessment. Is there a statistical test I could use run in SAS to compare concordance between these 2 researchers? Thanks so much for your help!

Example Data set (Actual sample size = 790)

Researcher 1 Researcher 2

subject id completeness_rating subject id completeness_rating

1001 Complete (coded as 3) 2001 Complete (coded as 3)

1002 Partial (coded as 2) 2002 Incomplete (coded as 1)

1003 Incomplete (coded as 1) 2003 Incomplete (coded as 1)

Re: Computing Inter-Rater Reliability - Different subjects are scored by different researchers

ballardw — Thu, 04 Jun 2020 20:14:45 GMT

Do you have pretty strong evidence that the two groups should have received a similar proportion of scores?

Chi-sq might be appropriate. But if you don't have reason to believe that the subjects distribution of scores should be similar the differences could be the subjects and not the researchers.

I would be tempted to run a quick Proc Freq requesting the chi-sq just to see.

Re: Computing Inter-Rater Reliability - Different subjects are scored by different researchers

Kimberly09 — Thu, 04 Jun 2020 22:46:44 GMT

Thank you for your reply. No, I don't have statistical evidence as to whether the 2 groups should have similar scores. However, the groups have similar demographic and clinical characteristics and the researchers seem to think the scores should be similar. I haven't done any statistical test to determine if this is the case (so far, I've only run the usual proc freq/univariate procedures). Here comes a basic question regarding the chi-square test.....Would I use the code below to determine if there's an association between the completeness variable and site (researchers), and if so, then what would I need to do?

proc freq data=check; 
tables complete_rate*site/chisq; /*the 2 researchers are at different sites/medical facilities*/
run;