Could proc freq give a warning when Cohen's kappa has zero standard er...

ceec · ‎11-29-2023

Sample code, based on SAS Usage note 22883 http://support.sas.com/kb/22/883.html

data a;
   do Rater1=1 to 2;
      do Rater2=1 to 2;
         input Wt @@;
         output;
      end; 
   end;
   datalines;
0 3
0 3
;
run;

proc freq data=a;
   weight Wt / zeros;
   tables Rater1*Rater2 / agree nocol norow nopercent plots=none;
run;

This code generates a 2x2 table to estimate Cohen's Kappa with standard error and confidence interval. One of the margins has a sum of zero, but it is forced to fit a 2x2 table using the weight statement and zeros option. The results are that the estimated Kappa is zero, the standard error is zero and the Confidence Interval is (0 to 0).

I've convinced myself that the standard error result is numerically correct, having checked the result using equations in the documentation and the cited reference. I think it is a special case for 2x2 tables, since the usage note above gives an example for 4x4 tables, and the resulting standard error is strictly positive. That said, an estimated statistic having no variability might not appear correct to a reader.

I suggest that

1) PROC FREQ provides a warning/note in the log/output to draw attention to this anomaly.

2) Ideally, if there are suitable improved formulae that don't have this defect, could they be implemented as an alternative option? (I admit, I don't have a better suggestion in mind).

Reference:

Fleiss JL, Cohen J, Everitt B. (1969) Large sample standard errors of kappa and weighted kappa. Psychological Bulletin. 72 (5) 323-327. https://api.semanticscholar.org/CorpusID:85541244, https://doi.org/10.1037/H0028106

Could proc freq give a warning when Cohen's kappa has zero standard error and zero-width CI?