topic Re: Comparing multi-rater kappas in Statistical Procedures

Comparing multi-rater kappas

shaz1 — Mon, 27 Jun 2016 14:30:15 GMT

Hi all,

I have two data sets where multiple raters have rated multiple x-rays into 3 categories. All raters have rated all x-rays and there is no missing data.

I've calculated overall kappas using magree for each of the two data sets. Is anyone aware of a way to statistically compare these two kappas? The macro produces a standard error for the overall kappa, which is very tiny (presumably because there are >200 raters in each group). Would it be proper to use this SE to create confidence intervals? For some reason the macro doesn't create confidence intervals, which makes me concerned I'm incorrect in doing this. Are there other ways to create these confidence intervals? I've seen one example of bootstrapping to create CIs in this situation as well - not sure which is most appropriate.

Thanks!

Re: Comparing multi-rater kappas

Rick_SAS — Mon, 27 Jun 2016 14:47:42 GMT

You mention a macro for computing the kappas. Is it the %MAGREE macro that is provided by SAS? Can you provide a link or reference for the bootstrapping example that you are considering?

Re: Comparing multi-rater kappas

shaz1 — Mon, 27 Jun 2016 14:52:36 GMT

Yes - the %magree macro from SAS is what I used.

This is what I've read about using bootstrapping to generate CIs for kappas:

http://support.sas.com/resources/papers/proceedings12/323-2012.pdf