Calcite | Level 5

## Trying to calculate AC1 statistic for multiple raters

Hello!  I am trying to calculate Gwet's AC1 statistic for multiple raters applying a scoring tool to multiple articles in a review.

I have been able to do a calculation for 2 raters using the example data and code below (2 RATER CODE).  In this example, there were 31 instances where two raters both scored an article a 6, two instances where they both scored an article a 5, two instances where Rater1 scored an article a 6 and Rater2 scored it a 5, and so on...

However, I also have data from testing this tool where there were multiple raters (n = 6 or 😎 scoring 4- 8 articles.  I have not been able to figure out how to code for the multiple raters in SAS.  I have included an example of what this data may look like below under "DATA FOR 6 RATERS".  When I try to use the same code for the agreement statistics for 2 raters (e.g. raterxx*rateryy*raterzz) it does not work.

Any help would be much appreciated!  Thanks!!

DATA/CODE FOR 2 RATERS:

data dataset;
input Rater1 Rater2 Count;
datalines;
6 6 31
5 5 2
6 5 2
6 4 1

4 5 1
;

data dataset2;
set dataset;
wgt = count;
run;

proc freq data=dataset2;
weight wgt / zeros;
test kappa;
tables rater1*rater2 / agree(AC1) nocol norow nopercent plots=agreeplot;
run;

DATA FOR 6 RATERS:

data dataset;
input Rater1 Rater2 Rater3 Rater4 Rater5 Rater6 Count;
datalines;
6 6 5 4 5 5 1
5 5 5 5 4 4 1
5 5 5 5 5 5 1
6 4 6 6 5 5 1

;

2 REPLIES 2
SAS Super FREQ

## Re: Trying to calculate AC1 statistic for multiple raters

The AC1 statistic is not available, but a multi-rater version of the kappa statistic is available with the MAGREE macro.

Calcite | Level 5