How to define own weights for the calculation of weighted kappa

[Jason2020]

Hello,

I have a large data set with several raters and I would like to calculate the intra-rater weighted kappa for each rater. However, I would like to define my own weights for the calculation of the weighted kappa. The Proc Freq documentation is not clear on whether this is possible or how to do it. I am including below a log of the analysis for a subsample, showing the frequencies, the kappa values, and the printout of the default weights and my desired weights. Any help is greatly appreciated.

Here is the sas code I used to generate these in SAS 9.4 (TS1M4):

proc freq data=amp;
tables PPD1 * PPD2/ nopercent norow nocol agree(printkwts);
weight Wt / zeros;
run;

Table of PPD1 by PPD2
PPD1	PPD2
Frequency	1	2	3	4	5	6	7	Total
1	36	19	1	1	0	0	0	57
2	21	39	10	4	0	0	0	74
3	4	17	13	4	0	0	0	38
4	1	3	3	3	0	0	0	10
5	0	1	2	0	0	0	0	3
6	0	1	0	0	0	0	0	1
7	0	0	0	0	0	0	0	0
Total	62	80	29	12	0	0	0	183

Kappa Statistics
Statistic	Value	ASE	95% Confidence Limits
Simple Kappa	0.2620	0.0533	0.1576	0.3664
Weighted Kappa	0.3584	0.0502	0.2601	0.4567

Kappa Coefficient Weights
PPD2	1	2	3	4	5	6	7
1	1.0000	0.8333	0.6667	0.5000	0.3333	0.1667	0.0000
2	0.8333	1.0000	0.8333	0.6667	0.5000	0.3333	0.1667
3	0.6667	0.8333	1.0000	0.8333	0.6667	0.5000	0.3333
4	0.5000	0.6667	0.8333	1.0000	0.8333	0.6667	0.5000
5	0.3333	0.5000	0.6667	0.8333	1.0000	0.8333	0.6667
6	0.1667	0.3333	0.5000	0.6667	0.8333	1.0000	0.8333
7	0.0000	0.1667	0.3333	0.5000	0.6667	0.8333	1.0000

This is my desired weights:

PPD2	1	2	3	4	5	6	7
1	1.0000	1.0000	0.7500	0.0000	0.0000	0.0000	0.0000
2	1.0000	1.0000	1.0000	0.7500	0.0000	0.0000	0.0000
3	0.7500	1.0000	1.0000	1.0000	0.7500	0.0000	0.0000
4	0.0000	0.7500	1.0000	1.0000	1.0000	0.7500	0.0000
5	0.0000	0.0000	0.7500	1.0000	1.0000	1.0000	0.7500
6	0.0000	0.0000	0.0000	0.7500	1.0000	1.0000	1.0000
7	0.0000	0.0000	0.0000	0.0000	0.7500	1.0000	1.0000

[StatDave · Accepted solution]

You can alter the weights used to some extent by selecting suitable numeric values for the levels of the column variable such that, when used with one of the weight formulas, produces desired weights. See the two selectable weight formulas for kappa in the Details: Statistical Computations: Tests and Measures of Agreement section of the FREQ documentation. However, I don't believe that the weights you want can be created with any selection of numeric column values. You could instead use Gwet's weighted agreement coefficient (AC2) available in the MAGREE macro. As described there, you can specify a weight matrix for use with that statistic.

The following creates the form of your data needed for the macro. Note that the macro is designed to deal with multiple raters, so the form of the input data set is not a simple a square matrix as used for the kappa statistic in PROC FREQ which is limited to the two rater case. See the first example in the macro documentation which shows the data set form needed.

data a;
   do row=1 to 7; do col=1 to 7;
   input f@@;
   do i=1 to f; 
    s+1; 
    r=1; y=row; output;
    r=2; y=col; output;
   end;
   end; end;
   datalines;
36      19      1       1       0       0       0
21      39      10      4       0       0       0
4       17      13      4       0       0       0
1       3       3       3       0       0       0
0       1       2       0       0       0       0
0       1       0       0       0       0       0
0       0       0       0       0       0       0
;

Note that the all-zero row and column in your data table cannot be used by the macro, so they are ignored and the resulting 6x6 table is analyzed. The following statements create the data set of weights for the AC2 statistic and then calls the MAGREE macro to compute the weight Gwet statistic. If you remove the WEIGHT=WT option, it provides the unweighted Gwet AC1 statistic.

data wt;
   input w1-w6;
   datalines;
   1.0000  1.0000  0.7500  0.0000  0.0000  0.0000
   1.0000  1.0000  1.0000  0.7500  0.0000  0.0000
   0.7500  1.0000  1.0000  1.0000  0.7500  0.0000
   0.0000  0.7500  1.0000  1.0000  1.0000  0.7500
   0.0000  0.0000  0.7500  1.0000  1.0000  1.0000
   0.0000  0.0000  0.0000  0.7500  1.0000  1.0000
;
%magree(data=a,
       items=s, raters=r, response=y, 
       stat=gwet, weight=wt, options=nosummary)

[Jason2020]

Thanks. This seems more complicated than I thought. And I ran into an error when I executed the code you sent. Below is the log with the error. It seems that sas cannot locate the macro.

Does this macro work for 2 raters, or does it require more than 2 raters? 

1 data a;
2 do row=1 to 7; do col=1 to 7;
3 input f@@;
4 do i=1 to f;
5 s+1;
6 r=1; y=row; output;
7 r=2; y=col; output;
8 end;
9 end; end;
10 datalines;

NOTE: SAS went to a new line when INPUT statement reached past the end of a line.
NOTE: The data set WORK.A has 366 observations and 7 variables.
NOTE: DATA statement used (Total process time):
real time 0.04 seconds
cpu time 0.04 seconds

18 ;
19
20
21 data wt;
22 input w1-w6;
23 datalines;

NOTE: The data set WORK.WT has 6 observations and 6 variables.
NOTE: DATA statement used (Total process time):
real time 0.01 seconds
cpu time 0.01 seconds

WARNING: Apparent invocation of macro MAGREE not resolved.
30 ;
31
32 run;
33
34 %magree(data=a,
-
180
ERROR 180-322: Statement is not valid or it is used out of proper order.

35 items=s, raters=r, response=y,
36 stat=gwet, weight=wt, options=nosummary);

[StatDave]

You need to read the macro documentation. This macro must be downloaded and defined in your SAS session. See the Usage section of the documentation. You should also read the Details section to understand the Gwet statistics and use of weights.