- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
Hello! I am trying to calculate Gwet's AC1 statistic for multiple raters applying a scoring tool to multiple articles in a review.
I have been able to do a calculation for 2 raters using the example data and code below (2 RATER CODE). In this example, there were 31 instances where two raters both scored an article a 6, two instances where they both scored an article a 5, two instances where Rater1 scored an article a 6 and Rater2 scored it a 5, and so on...
However, I also have data from testing this tool where there were multiple raters (n = 6 or 😎 scoring 4- 8 articles. I have not been able to figure out how to code for the multiple raters in SAS. I have included an example of what this data may look like below under "DATA FOR 6 RATERS". When I try to use the same code for the agreement statistics for 2 raters (e.g. raterxx*rateryy*raterzz) it does not work.
Any help would be much appreciated! Thanks!!
DATA/CODE FOR 2 RATERS:
data dataset;
input Rater1 Rater2 Count;
datalines;
6 6 31
5 5 2
6 5 2
6 4 1
4 5 1
;
data dataset2;
set dataset;
wgt = count;
run;
proc freq data=dataset2;
weight wgt / zeros;
test kappa;
tables rater1*rater2 / agree(AC1) nocol norow nopercent plots=agreeplot;
run;
DATA FOR 6 RATERS:
data dataset;
input Rater1 Rater2 Rater3 Rater4 Rater5 Rater6 Count;
datalines;
6 6 5 4 5 5 1
5 5 5 5 4 4 1
5 5 5 5 5 5 1
6 4 6 6 5 5 1
;
- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
The AC1 statistic is not available, but a multi-rater version of the kappa statistic is available with the MAGREE macro.
- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
Many thanks for your response!
I have not previously used macros, though I did try to use a macro I found online for multiple-rater Gwet's AC1 that did not work. I am using SAS UE, and not sure if that influences things. I will give this a try.