topic Re: Confidence interval Observed Agreement in Kappa in Statistical Procedures

Confidence interval Observed Agreement in Kappa

dcalbet01 — Fri, 18 Dec 2020 13:07:12 GMT

Hello

I am trying to calculate the confidence interval for the agreement observed in the Kappa analysis.

If I run for example

data FatComp;
input Test Response Count;
datalines;
0 0 6
0 1 2
1 0 4
1 1 11
;
proc sort data=FatComp;
by descending Test descending Response;
run;
proc freq data=FatComp order=data;
weight Count;
tables Test*Response / AGREE(KAPPADETAILS) ;
run;

I get that for the Kappa statistic, I get the confidence interval. But I would also like to obtain the confidence interval for the observed agreement (in Kappa Details 0.7391).

First I thought it would be like the confidence interval of a proportion (P + -1.96 * SQR (p (1-p) / N), but it isn't ... any ideas?

Thanks ¡

Re: Confidence interval Observed Agreement in Kappa

SteveDenham — Fri, 18 Dec 2020 14:14:08 GMT

The point estimate of observed agreement is fixed by the row and column totals. I don't see any easy way to calculate the standard error based on this other than the method you propose. You have the estimate for p, so now it boils down to what N to use. I would be tempted to use the sum of the diagonals (N=17) which is the agreement numerator. From that I get, LB=0.7391 - 1.96 * sqrt (0.7391 * 0.2609/17) = 0.5304. For the UB I get 0.7391 + 1.96*(0.7391*0.2609/17) = 0.9479. Does this make sense? The interval would be narrower for N as the total = (0.5597, 0.9186).

SteveDenham

Re: Confidence interval Observed Agreement in Kappa

Watts — Fri, 18 Dec 2020 16:36:58 GMT

Observed agreement (the sum of the diagonal proportions in the crosstabs table) is also referred to as accuracy. This SAS Usage Note (Estimating sensitivity, specificity, positive and negative predictive values, and other statistics) shows how to create the corresponding binary variable for accuracy and then use the BINOMIAL option to get confidence limits and tests. From the usage note,

data acc; 
   set FatComp;
   acc = (test = response); 
   run;
 proc freq;
    tables acc / binomial(level="1");
    weight count;
    run;

You can also use the BINOMIAL(CL=type) option to specify the confidence limit type (for example, Agresti-Coull, Wilson, etc.).

For 2x2 tables, the SENSPEC option (in the PROC FREQ TABLES statement) now includes accuracy together with confidence limits in the Sensitivity and Specificity table.