BookmarkSubscribeRSS Feed
CV9
Calcite | Level 5 CV9
Calcite | Level 5

Hello!  I am trying to calculate Gwet's AC1 statistic for multiple raters applying a scoring tool to multiple articles in a review.  

 

I have been able to do a calculation for 2 raters using the example data and code below (2 RATER CODE).  In this example, there were 31 instances where two raters both scored an article a 6, two instances where they both scored an article a 5, two instances where Rater1 scored an article a 6 and Rater2 scored it a 5, and so on...

 

However, I also have data from testing this tool where there were multiple raters (n = 6 or 😎 scoring 4- 8 articles.  I have not been able to figure out how to code for the multiple raters in SAS.  I have included an example of what this data may look like below under "DATA FOR 6 RATERS".  When I try to use the same code for the agreement statistics for 2 raters (e.g. raterxx*rateryy*raterzz) it does not work.  

 

Any help would be much appreciated!  Thanks!!

 

 

DATA/CODE FOR 2 RATERS:

 

data dataset;
input Rater1 Rater2 Count;
datalines;
6 6 31
5 5 2
6 5 2
6 4 1

4 5 1
;

 

data dataset2;
set dataset;
wgt = count;
run;

 

proc freq data=dataset2;
weight wgt / zeros;
test kappa;
tables rater1*rater2 / agree(AC1) nocol norow nopercent plots=agreeplot;
run;

 

 

DATA FOR 6 RATERS:

 

data dataset;
input Rater1 Rater2 Rater3 Rater4 Rater5 Rater6 Count;
datalines;
6 6 5 4 5 5 1 
5 5 5 5 4 4 1
5 5 5 5 5 5 1
6 4 6 6 5 5 1

;

 

 

 

2 REPLIES 2
StatDave
SAS Super FREQ

The AC1 statistic is not available, but a multi-rater version of the kappa statistic is available with the MAGREE macro.

CV9
Calcite | Level 5 CV9
Calcite | Level 5

Many thanks for your response!  

 

I have not previously used macros, though I did try to use a macro I found online for multiple-rater Gwet's AC1 that did not work.  I am using SAS UE, and not sure if that influences things.  I will give this a try.  

SAS Innovate 2025: Call for Content

Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!

Submit your idea!

What is ANOVA?

ANOVA, or Analysis Of Variance, is used to compare the averages or means of two or more populations to better understand how they differ. Watch this tutorial for more.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 2 replies
  • 1039 views
  • 0 likes
  • 2 in conversation