topic Re: Please help new user! Interobserver and intraobserver reliability in New SAS User

Please help new user! Interobserver and intraobserver reliability

ancfletcher — Mon, 05 Aug 2019 13:15:01 GMT

Good morning SAS community,

I have what I believe to be a relatively simple question regarding intraobserver and interobserver reliability using a weighted Cohen’s kappa analysis.

I have two reviewers (R1multiall, R3multiall) that graded radiographs 1-3.

R1multiall	R3multiall
1	3
3	2
3	3
3	3
3	1
3	2
3	2
1	1
3	2
2	1
3	1
3	3
1	1
3	1
2	1
2	3
3	1
3	3
3	3
3	3
3	2
3	3
1	1
3	3
3	1
3	1
3	2
1	1
2	1
2	1

In trying to determine the inter-rater reliability, I used the code:

proc freq data = scaphoid.scaphoid ;
tables R1multiall * R3multiall / agree ;
test kappa ;
run;

However, I am surprised by the results (weighted kappa 0.1935) and feel that I am not calculating this correctly. I apologize, I am new to SAS and would really appreciate some help! This is just a small part of my dataset, I actually have 5 reviewers who all reviewed the radiographs twice. So I would like to access both inter- and intra-rater reliability.

Thank you for any guidance!

Re: Please help new user! Interobserver and intraobserver reliability

PGStats — Mon, 05 Aug 2019 18:16:25 GMT

Why are you surprised by the results?

Re: Please help new user! Interobserver and intraobserver reliability

ancfletcher — Mon, 05 Aug 2019 18:31:24 GMT

The kappa correlation coeff for the example above (12 scores in agreeance) was 0.19. versus this new dataset below (17 scores in agreance) with a kappa coronation coeff of 0.02. I know the weighted kappa accounts for the ordinal nature of the data which could be having more of an influence than I expected. Do you think using ICC would be better in this scenario given the rank of the scores?

R1MultiAll	R4MultiALL
1	3
3	3
3	3
3	3
3	1
3	3
3	3
1	3
3	3
2	2
3	3
3	3
1	3
3	1
2	3
2	3
3	3
3	3
3	3
3	3
3	3
3	3
1	3
3	3
3	2
3	3
3	3
1	3
2	1
2	3