Trying to calculate AC1 statistic for multiple raters

CV9 — Wed, 15 May 2019 02:30:09 GMT

Hello! I am trying to calculate Gwet's AC1 statistic for multiple raters applying a scoring tool to multiple articles in a review.

I have been able to do a calculation for 2 raters using the example data and code below (2 RATER CODE). In this example, there were 31 instances where two raters both scored an article a 6, two instances where they both scored an article a 5, two instances where Rater1 scored an article a 6 and Rater2 scored it a 5, and so on...

However, I also have data from testing this tool where there were multiple raters (n = 6 or 😎 scoring 4- 8 articles. I have not been able to figure out how to code for the multiple raters in SAS. I have included an example of what this data may look like below under "DATA FOR 6 RATERS". When I try to use the same code for the agreement statistics for 2 raters (e.g. raterxx*rateryy*raterzz) it does not work.

Any help would be much appreciated! Thanks!!

DATA/CODE FOR 2 RATERS:

data dataset;
input Rater1 Rater2 Count;
datalines;
6 6 31
5 5 2
6 5 2
6 4 1

4 5 1
;

data dataset2;
set dataset;
wgt = count;
run;

proc freq data=dataset2;
weight wgt / zeros;
test kappa;
tables rater1*rater2 / agree(AC1) nocol norow nopercent plots=agreeplot;
run;

DATA FOR 6 RATERS:

data dataset;
input Rater1 Rater2 Rater3 Rater4 Rater5 Rater6 Count;
datalines;
6 6 5 4 5 5 1
5 5 5 5 4 4 1
5 5 5 5 5 5 1
6 4 6 6 5 5 1

;

Re: Trying to calculate AC1 statistic for multiple raters

StatDave — Tue, 21 May 2019 13:26:34 GMT

The AC1 statistic is not available, but a multi-rater version of the kappa statistic is available with the MAGREE macro.

Re: Trying to calculate AC1 statistic for multiple raters

CV9 — Thu, 23 May 2019 01:23:31 GMT

Many thanks for your response!

I have not previously used macros, though I did try to use a macro I found online for multiple-rater Gwet's AC1 that did not work. I am using SAS UE, and not sure if that influences things. I will give this a try.

topic Trying to calculate AC1 statistic for multiple raters in Statistical Procedures

Trying to calculate AC1 statistic for multiple raters

Re: Trying to calculate AC1 statistic for multiple raters

Re: Trying to calculate AC1 statistic for multiple raters